Live CEOing Ep 816: Language Design in Wolfram Language [Tabular]

Hi, everyone. Welcome to another Wolfram Language Design review about tabular. And let's see where we're starting from.

So we were discussing missing. And there's a section there called SW Nodes. So that's a reminder of what we had done last week. So. Okay.

By the way, I just want to comment that failsafe really wants to get into 12 for 14.1. And I think that's. I don't know whether Christopher has stuff to do on that or what. Who has.

I think. I think we were gonna. We're gonna meet in some LCC meeting to talk about basically how it should handle options. That was my understanding.

So is that pending? Because, I mean, it needs to be really wrapped up for 14.1. So.

Yeah, basically, if the answer is it doesn't handle options, then it's totally wrapped. And I think it probably shouldn't handle options.

But if it should handle optical options, what does it mean to do that?

So, okay, I mean. Right. I mean, maybe you shouldn't have the whole discussion here, but basically, it's. It's. If you, you know, you say fail, Fail Safe is like a headdress. If you say Fail Safe of, you know, plot. Not that you'd want to do that, of xyz, that's effectively a way of running plot of xyz. But if any of X, Y and Z are missing or broken or whatever, it will fail more gracefully. So the question is, what if you say Fail Safe plot of XYZ comma plot range goes to something rather. Should it inspect the plot range goes to something rather in some intelligent way.

We're not going to get that in for this version. Let's get a version of Fail Safe that does not do that, and we can worry about it doing that later.

Okay, well, there already is a version that doesn't do that. So.

Yeah, and remind me what it does. So failsafe, just. If there's a message or something else that goes wrong, what does it return?

Not a. I don't think it detects messages, but it's basically. I mean, I guess maybe it. It could, but. No, I mean, failsafe is basically about. It's basically about if any of the arguments are broken in some way, then it will propagate those broken results up.

So in other words, if there's a failure in one of the. If the function to be plotted is a failure object, then it will propagate that. Is that correct?

Right. I mean, plot's not a good example. Better example, like, the simplest thing would be something like fail safe of plus of one comma missing will return missing.

I see.

And in general, it has their other arguments to the headdress for if you want to detect kind of more complicated cases.

Okay, okay, so let's, let's go back to what we were talking about before.

Sorry, administrative thing. I don't think we have an LCC meeting next week, so I don't know how much discussion this require.

I think we just. I think we just finished it. I mean, I think we decided what we want to do, though.

Okay, fine.

That's great.

That's great. Okay.

Yes. So the main proposal last week, and I think Carlo and Rob can explain better, was to separate the behavior for column wise operations and element wise operations with respect to missing handling in the sense of being more permissive, like ignoring missing by default in column wise operations, but not in element wise operations. And then we had decided to have this function if missing that allows control on what to do if you find a missing in an operation that cannot or should not.

If missing is still a very tentative name because it's, it does. It's not really like if.

I'm. So I'm sorry. Okay, let's go through this in a little bit more detail. So first point is the default behavior is that in a column wise operation, a missing like object is ignored. Is that correct?

Yes. In aggregation operations, yes.

Yep.

So that means that in construct columns, for example, this is not needed. Is that correct?

Well, that's an, that's an example of an element wise operation that goes row by row. Then the example below in tabular reduce would be an example in which mean would ignore missings. But if we want not to ignore them, we would introduce if missing.

That's like a hyperlocal version of replace missings, right? So that, you know, depending on whether it's on the numerator or on the denominator, you. You select a different default, for example.

I understand. How does it do us any good to insert if missing if the missings are already gone? No, go ahead.

Okay, so this is slightly different than what you would get from. From just the normal propagation of missings, because basically if you divide hash A and B, you would get a missing result anytime either A or B is missing and also anytime B is 0. Instead this will give you a different result because if just B is missing, it will replace it with one instead of replacing the whole thing with missing.

I understand it. I'm asking how do you get out of the default? If the default.

This is getting out of the Default.

How, how does it work? Because the, the if there's a column, presumably before we start doing anything on the column, we are deleting all the missing.

No, no, no, no.

Because.

Because the division is listable. And so before we even get to the dropping of the missings, we perform the operations on the two sides of divide element wise. So it's the mean that is the column wise operation. The other operations are performed row wise because they thread, they list.

Okay, so let's go through this again. Okay, so are we proposing basically one function here, plus perhaps some options controlling row and column default behavior? Is that what we're talking about?

Yes. So those options are in the global control section below. It's something like acts globally with something similar, not necessarily identical, but similar to failure action or missing behavior with some dollar defaults. And then locally we control with if missing what to do.

Okay, so. Okay, so the if missing thing, which is a bad name, as you say. What was the idea with test applied?

Test applied is an old name for failsafe.

Okay, okay. And what was this idea here, this not valid idea?

I think that was about if you. Right. So there was discussion of having like an option or something for tabular columns so that you could tell it that, for example, you know, maybe minus 99 in the inputted data should be interpreted as missing. So the question is, if you index into that, if there's a minus 99 there, should it come back as minus 99 or should it come back as missing? Or it should come back as a minus 99 marked that it is a kind of missing.

Like. Yes.

The concept of missing may be a little bit misleading. It's more like invalid.

I know, which is why, that's why, that's why we're suggesting not valid.

Exactly, exactly.

Yeah. I mean, and, and this would be. Let's see, This wrapper here wants to be something like valid. What the heck is it? It's. It's. It's not really. Right. I mean, it's kind of like it is expert. Unless it's the. Unless it's, you know, unless it's the default, so to speak. If exper is not valid, then it's the default. I mean, maybe it's. It's more like check valid. And then there's a wrapper not valid, which we can put around things, which is not the same as missing. But check valid would. I don't know whether valid is the right word for it, but you get what I'm saying. Check. What is the point? It's Whether it's to be included or not. What is the real operate? What is the real thing we're doing here?

Well, I would say the key separation is whether one, we want to ignore it as if it wasn't there, or we want to replace it with something else or even propagate it.

Propagate it? Yeah.

Well, so what does it mean to propagate it? I mean, to propagate it is just. Do you want to wrap that expert in something?

Just leave it there. Even if we are going to get a 10 plus missing. 10 plus three missings.

Yeah. Okay, hold on. So what is it doing? Explain in one sentence what this thing is doing.

When you say this thing, can you.

At least this function check valid or whatever it's called, it basically. Returns. I mean, if EXPA is invalid in some way, return default, otherwise return expert, right? Yeah. Correct.

Yeah, I think so, yeah.

And default can presumably be. I mean, I guess nothing, Right? For the case that removes it.

Yes. Also check valid. What's that?

Perhaps it could be even the value of another column or something like that.

Well, but okay, but hold on. If, if check valid of expo with one argument might be equivalent to check valid of expo comma. Nothing. I mean, I worry that check valid is such a generic name.

Yeah.

Are you saying that check valid is the replacement for if missing?

Yes.

No, that won't work because it has to be Vargas. Look at the example with species, genus, family and not available. Yeah, that's very like. This is a cascading operation where you. You try one thing, if that fails, try the next. If that fails, try the next. It's really way more useful that way.

Okay, fine, fine. So that's not. I see. Okay, so. Okay, so let's. That was what I'm trying to get somebody to tell me.

Yeah, but I didn't understand what check valleys was.

Okay, so we need a name for this function here.

Yeah.

Could be something like fall through or something.

That's better, but not quite right in my view. Let's see what else is.

Actually, Also a quick question. The quotes not available, are you imagining it returned the string not available or missing of not available?

Yeah, no, the string not available string. Yeah, that's all literal. Like I'm thinking about, like you're building a description string and you can't. Maybe you want to string join it with something. You can't string join it to missing. And so you're checking, trying to find things in various columns. And if you don't, then provide a default. Or maybe you're satisfied that family is never missing. So you don't need the last one. You put whatever you want there.

Yeah.

Fallback.

That's not a terrible name.

I don't think it's terrible. Yeah, Fall Back. This is not really right, but something like that. Well, so, okay, so the idea for Fallback would be that Fallback keeps going in this list until it reaches something. Until either reaches the end of the list or it reaches something which is not a failure case. Correct?

Yes.

Is that correct, guys?

Yes. Yes.

Y. That's correct.

Falls back to later items, you know, returns the first item, first of the X per I. That is not invalid. Not a failure of some kind.

Yeah, I actually wouldn't use invalid really or failure. This is really more useful for missings. Like to me it feels that if there's a failure in there somewhere, then the whole thing should fail. Missing.

Oh, no, but that's actually, I think Stephen's thinking a little more generally because, you know, like, suppose you have a symbolic calculation and your first one is a. Is a good algorithm, but it's only very specific cases, so that would fail. So then you go to the second. The second option and so on. So I mean, there's probably needs to be some kind of a failure pattern or missing pattern, you know, to decide. But, but this is actually as a name, Fallback. It's actually a very. It's a very general idea and we can make it very efficient specifically for missing. But, but I kind of like the name in terms of the general concept.

So would you. Would we have any. Yeah, right.

Because we do that all the time in numerical stuff. You know, if, if, you know, the runga cutter method is not going to work because the equation's stiff, well, then we go to the, you know, the next method and so on and so forth.

I see.

So then I just worry about, you know, discoverability for people who come from data science kind of workflows.

Couldn't care less.

Sure.

That's.

I mean, it's a different system. It's. What are they. What do they expect it to be called? What do they expect it to be called?

Coalesce. Both SQL and Pandas use coalesce. Okay, I was talking more about like, or something that says missing. No, not something that says failure.

Okay. That's a documentation tagging issue.

Yeah.

Fallback is also. Carl, you had indicative than. Than coalesce.

Yeah. Carla, you had some special list behavior too. So if exper was a list. Right.

That did well.

You know, I think it, you know, in light of the example with the tableau reduce above. I think that, you know, it's better if we keep it clean for now.

Yeah, I think we should keep it clean and never. Because I think this is a useful general construct.

Yep, I agree.

Good.

Because I mean basically it means that you can just basically throw. I'm just wondering how that would work. You could. Yeah, you can just say catch of expert one and somewhere deep inside it you can throw a failure. I'm thinking about general uses of this.

Yep, that's. That's exactly, that's exactly how you'd use it. I mean, I wonder maybe we should have a missing fallback as a special case just for, for missing. That could be done more efficiently and that would satisfy Carlo's issue about the missing.

Well, I'm not sure but the question.

Is really also what happens if all of them fail? I. I don't, I don't.

I mean, so we have. Then it gives a failure object. I would assume if all of them fail, it gives a failure object.

Well, does it give a failure object or does it give the last one like the example above where it's got the quotes. I guess quotes not available is available. So.

Yeah, yeah. I don't know. I mean in terms of what it does in other languages, if they all are null, then it just gives null. So if they translate it to us, if they're all missing, it returns missing.

Yeah, well, it's a little bit like switch or which without a catch all case.

Yeah, but look, okay, we got a couple of issues here. One is if there's anything special for missing. The other thing is what does fallback. What is the. What's the. There are other functions like confirm that has the analog of a failure pattern. Right.

Well, there's confirm match, which just the second argument is a failure pattern basically.

But what is. What does confirm itself? Does it have an option or does it just have a particular set of.

There's other fun. If you want to specify the failure behavior, you use Confirm buy or confirm match.

I see.

I think we talked about a while ago though, I mean this is related. Is having a second argument for failure queue where you would be able to specify like where we could have named classes of failure like objects.

Yes.

So I mean you'd imagine that would be the value of this. That would be one thing you could put on the right hand side of this would be.

Well, that's, that's. That's the current focus of their handling working group.

So.

Okay, great.

There's a lot of work Being done on registering classes of failures and so on. I think maybe Leonid can speak more to it if he wants to.

Yeah, I mean, if, if we want to discuss it now, I can.

Let's not discuss it now. I just want to know the bottom line here, that that is going to be some appropriate. You know, there's this thing here, could be a fallback with a, you know, some kind of failure failure test, Which seems more general than a failure pattern. I don't maybe, I don't know. What do you think? Failure pattern, failure test, Maybe both of them.

I think if it's failure, it's neither. The point is, okay, the Error Handling Working Group is building some very specific things about, like defining classes of failures so that in general you wouldn't use neither patterns nor tests for failure. But, but if this thing, by failure it means anything that is missing queue, then maybe it should be a test and that test should be missing queue by default. That's what I kind of don't like about failure tests is that at least in this context, you want that list of invalid things where failure is just one little thing. If it concentrates on failures, we have better tools for failures.

Wait, so would you want it to default to only doing missing or to do all kinds of missing failure?

Right now, failure is missing queue. I think.

I think failure test is a good name if the default is failure queue. This reminds me of same test defaulting to SameQ in general. But we can use. We can use equal.

Yeah, that's a good point.

Yeah, exactly.

So let's just see. Failure queue of missing, blah, blah, blah.

Yeah, you see?

Well, that's no good.

This is why we talked about.

I mean, I think last time we.

We talked about this a while ago, and the idea was that basically this is probably correct behavior, but that there should be a second argument for failure queue where you could tell it about a different class of failures that you care about.

Okay, but then what should this be? So, I mean, the question is, this is an option. I assume there's no kind of headdressy type thing that says something like, you know, Test and then Expo 1. I assume that makes no sense. That would be a different design structure. I think it makes sense, but why not?

It looks cleaner.

Well, but you now always have to provide the test.

And presumably we want this to be hold Rest.

Yeah, yeah, hold. Well, yeah, hold all. I would say it's got to be hold all.

I mean, unless we put the test first.

Yeah, well, no, but you're always going to evaluate the first one, Right?

Presumably, yes, but. But look, I think we just need a name, okay? This name is the same name that we're going to need for the thing that says not valid for the wrapper that says not valid. See what I'm saying? It's a generic name for. A thing that will be omitted. Is that correct?

I don't know if it's correct to conflate invalid with omitted like.

No, no, no, no, no, no. What we're trying to clarify is, for these purposes, what is the operational definition of the thing that we want to wrap the -99 with? I'm saying, what is operationally going on? I think it's that it's going to be dropped in various operations. Do you see what I'm saying? It doesn't matter whether there's a value judgment about whether it's valid or wonderful or not wonderful or shocking, whatever. What matters operationally is it's going to be dropped. Is that a true statement or not?

Yes, but I don't understand. What is it you don't like about missing, which has the ability to wrap things like you can have missing with two arguments and put something in the second argument.

So you're proposing that the -99 gets wrapped with missing.

Yeah, and it can be missing, invalid comma, minus 99, if you want.

Okay, so we don't need anything other than. So there's going to be an auto wrapping with missing. So what was the thing that we wanted to put in that? What's the thing? That's the option for a column that says what the missing pattern is.

Missing value pattern.

Okay, so then that would be missing value pattern, and that would say -99. But then what does it do with the missing value pattern? Does it wrap it with missing or what does it do?

Well, when it sees a -99, what it would do is it would. I think Carlo's suggestion is perfectly reasonable is, you know, it would do missing some tag which might be, you know, not valid or not parsed and then comma -99. A little bit like that. That's a little bit like the key. Absent ones.

Yeah. Okay. All right, so. So maybe what this needs to say is, I mean, I think it would be something like, you know, it might as well be very explicit. Excluded by pattern or something. Something like that. There's no reason to not be quite explicit. So if you see it.

Yeah, that's.

I'm glad.

That's actually much better than.

Yeah.

Okay. All right, so then let's understand this. So Then the missing value pattern applies to a column. Now as we go through the by default in a column. So I'm a little bit confused here, because there's also default missing value here. So is it also underscore missing? You see what I'm saying?

Yes, I think you'd have to. You'd have to make an alternative or, you know, some pattern that catches everything you want.

Okay, okay, fine. All right. Okay. So then those things get. Well, okay, if that's the case. Right. We could also have failure there. So in other words, there will be nothing. The only thing that would be treated as missing is a missing wrapper, because missing value pattern would even wrap a failure with a missing. Do you see what I'm saying? Yep, yep. So then that would become, you know, something like excluded by pattern failure, which I don't think is bad. I think then there's a single thing that we're dropping out, which is missings.

And missing itself would stay the same. So that wouldn't be mapped by this. Which is. Which is fine. I think that's great.

So wait a minute. So you're saying a missing value, that this is now irrelevant?

Well, no, it's still. It's detected, but it's not gonna. It's not gonna map to something different than the. The missing that was given.

Fine, but the defaults would be. Okay, but listen, if I say missing value pattern -99, omit the underscore missing. If there is a missing already there, it will just go through as a missing. Which is correct.

Yeah, that won't get rewrapped, which is correct.

Right. Okay. Yeah, that's okay. That's fine.

But the other thing I wanted to say is, like, in the work we've done on entities like, the missings are the things that appear in a given value, and the failures are when the whole thing goes completely crazy. So instead of returning a tableau, you return a failure.

I didn't know. Is that relevant to what we're talking about right now?

It's relevant. There is a section below about this which is like when. When the failure is so catastrophic that you need to.

Okay, let's not deal with that right now. So then the question is, you know, this could be called for now, missing fallback, Which is actually quite a nice, clear name, in my opinion. Yeah. And then we wouldn't have any test. Any options for testing. It would just be if it's got a missing, then fallback makes sense.

Yep, yep, yep.

Okay. All right, what's next here?

I guess one Question about listening. Fallback. And this, this relates to the discussion. Is it, is it listable or not?

I hope not. What does it have to be?

Well, suppose you give. Suppose the first argument is a list with a bunch of missings or some missings.

I think that's a total mess. I mean, I think, I think we should just not have it called in that case.

Yeah, but in some sense, if you look at the example above in Tableau reduce.

Yeah, that's how we're going to use it there.

And it's super useful to have that. So maybe it's listable, but in a special sense, in general, whenever you have something like imagine instead of having it missing and division, you could have mean of a plus one. In that case, that plus is listable.

Yeah.

Okay, so let me just understand what you're saying. So if you say it's one missing comma two.

Yeah.

And then say what's that? Comma zero. Okay, I see. And then, so then that would turn into 1, 0, 2, correct.

Right, exactly. And so see, that's a, that's an easy way to change the. Change the behavior for mean. So, so the, you know the case. If you put nothing, then that's the default behavior we're talking about. But if you put zero, then it's including that element, but as a zero. So the divisor is different in mean.

But I think it can't be really listable. It has to be like threadable because.

You could have a special thread 3.

List in any of the position of the arguments.

Right.

Okay, so hold on, let's look at what this could do. So let's, let's say we do this and then the second argument. I mean, this could get really weird. It's 4,5 missing. What the hell does that mean?

That means 1, 1, 5 2.

Yeah, 1, 5 2. Right?

Yeah, 1, 5 2. So I thought thread without an option. We can. I think, I think actually listable is almost whether it's.

Whether it's. But it's custom listability of some kind. Probably. Yeah.

Because it should fail. It should fail if one one list is like length four.

Yeah.

Oh, absolutely. But see, that won't happen. Listability is it. It gives us the failure automatically. I mean, it's a generic to generic failure then.

Right? Okay, so let's walk through this now. Okay, so then in an individual element, we are so missing value pattern just says how to remap. So wait a minute, how do you do missing value pattern per column? I see. I assume what we do is that the, the option could be missing value pattern, arrow association column one, arrow blah blah blah, column two, arrow blah blah blah.

Well, actually, what is the convention right now? Because it seems like there are a lot of options like missing value pattern that ultimately should be an option for tabular column, but where you might want to provide them inside the full tabular is the convention just that you can write something like this and it will distribute it.

Yeah, well, for example, the, the column types is done that way. It, you write it as an association, distributes it. Now the tabular column, we can have that as an option yet right now just to, just to, to save, you know, myself huge time changing options and processing options. It, it uses. So basically what tabular processing does and it converts that into an association of directives. Like so it has quote missing value pattern goes to blah and then that's passed to the pass to the individual tabular column constructor. But it could, it could be an option as well.

It's just. So right now it wouldn't be. I mean, but ultimately, I mean, is there a reason why this wouldn't be also an option to tabular column?

No, I think it can.

Yeah.

Yeah, it can.

Okay. Okay.

Yeah. Right. Okay. So either global option to tabular or option to tab it up column, right?

Yep.

But this is something at construction time, right?

Yeah.

And we might need, you know, I, I don't know if P's here or not, but he might comment on whether we need that at import time too.

I would think we do. Right. So what I committed actually a few hours ago to the prototype build is that there is a missing value pattern for import of CSV files that accepts well, string list of strings or string expression or regular expression. So basically everything string match queue works with. But this is global, not per column yet, but obviously I can do per column. Yeah, I think for CSV, it's probably global is going to be typically the thing one wants. Good. Okay, so let's just continue this review here and see if there's anything else we need for missing propagation. Or is this basically what we need for right now?

Well, the one, the, the one other question is, is, you know, is. And this, this sort of goes with the failure thing. So the example I brought up a while ago with say a list. So this is a list of strings, you know, most of which are dates. And you want to convert those to dates. What do we, we, you know, what do we call the missing. We generate, you know, when one of them is not a date.

Missing. Why isn't it just missing missing. You know, something like missing not. Not a date. I mean.

Yeah, fair enough.

Comma. Whatever the hell it is.

Okay, yeah, but what if they're all not dates? Is that a. Is that considered a separate case or not? Or do we let the user handle that?

I would just say missing not a date for every single one of them.

Fine.

Anybody see the problem with that?

No, I think it's fine. I think entity value does that too. It has lists of all missions if needed. I mean, it doesn't convert into a single missing.

Yeah, and also I think that if you really want to put a constraint that results in a hard failure, it should be possible to do. But other than that, there are only two choices. Either hard constraint or all things are being replaced by not a date or something like that.

Okay, good.

What is that convention for what the actual replacement is? What would be the. What are we calling it? Not date? Or do we have any other or invalid date?

But Steven, wouldn't it be the case that there will be not just dates? So there might be a need for more general convention, like not convertible or something that applies to other things where there is a mismatch of that kind.

Yeah, fine, but I mean.

Like, not like.

I understand, but we need to have a convention for what all of them are going to be. If it's not a date, if it's not a number, whatever. What does it do in interpreter? Carlo, you certainly know this.

Failure interpreter returns a failure if it.

Can'T parse, but its tag is interpretation failure.

Yes.

And if on the other hand, you put something like that doesn't need to be interpreted like you put the A number 2.9, then it says restriction failure.

I think.

No, not.

I mean, you need to put a restriction and blah, blah, blah.

Okay, fine, but then what is our convention here?

Maybe conversion would be good.

Yeah.

As far as I know, entity value doesn't tag things with types. It tends to say directly not available or not applicable.

Yes, I agree. What about something like that? If the conversion failure is the main thing, then we could just say date conversion failure if we happen to know it's supposed to be a date. Does that make sense?

I wouldn't really specify specific types because.

Okay, then let's not generally do that.

Just conversion failure would be, I think, more useful.

Okay, yeah, if we really wanted to do that, we could have the second argument missing be, you know, an arrow with the to type and the from type. But I think that's getting more detailed than.

Well, I don't Think. I think we want the second argument to the missing to be what the actual payload was. Mm. Okay.

Yeah.

Okay. What else do we have to figure out in this section? This is looking good to me. Can we make sure we write it up with. With all the crud removed? Yes.

Yep, I am.

Okay. And now this thing about this dollar failure action. Would we have a global missing value pattern? I don't know how necessary that is. A dolling dollar missing value pattern. Do we need that really? It seems wrong to me because it appears in lots of functions. I just don't think that's necessary.

Yeah, I think it would be a very easy way to shoot you in the foot.

Indeed.

It should at the very least be a local option, not a global setting.

Okay, so it is a local option. Missing value pattern is a local option to tabular and to tabular and so on. So we're good. Okay. All right, let's continue to the next section. Are we ready to go to the next section?

Well, that section was a bit of the let's study the precedents.

No, no, wait. There is another thing we need to talk about in this section, which is.

Please, let's talk about.

Okay, so the question is like. So I'll tell you what we do in entities. So, for example, if you have something like a local one over zero that is value dependent, there's no need to fail the whole operation. You just put a missing in the place where that division failed. But there are situations where you know straight away that you're trying to do like an addition between a string column and a numeric column. In that case, we usually fail spectacularly and return a failure because we just can't compile the query. And so the question is, do we want to have some kind of way to interpolate between those two behaviors and figure out. And this ties in also the question about the fallback evaluation. Maybe you want to stay into some strict mode where you never tried to fall back into slow evaluation and use just the kernel. And so maybe you want to turn on some kind of pedantic flag that everything fails unless it's fast, then that.

Will be a global option to tabular, presumably.

Yeah.

So the question is, like, how do you modulate? What do you want to fail? Like, if you want to construct a column full of missings whenever you can compile and go on, or you want to build a big failure object that's.

Presumably a failure action thing. Right?

Well, failure action, yes.

Yes and no. I mean, an example is, suppose you have a column of a bunch of reals and you do the exponential function on them. Do you want to. Do you want to fall back to big nums or generate missings, for example?

I don't think we know yet. Okay. And I recommend that what we do is if we have a bunch of cases that we allow in tabular a failure action, which can be a, you know, some complicated association which has a bunch of these cases, that's what I recommend.

Fail. Failure action being an association where you say, like, this goes. Falls back to normal evaluation.

Fail.

This is an error of one value put. Missing.

This is.

Yeah, that makes sense.

Like, because column operation. Column. Incompatible columns, let's say whatever. I don't know, incompatible columns. And that would be. Oops. And that would be, you know, failure object or something. I don't know. I'm not sure. But I'm just saying parameterize it in a way, because I don't think we really know what this is going to look like. And I think we should put in a bunch of different cases that we think of and just stack them all up in an association, And then later on we might clean it once we know matters. Okay. All right, let's keep going, please. I. I think we've got to get to pivots and things like this. But what. What about value cleaning? What's this? Is this what. We already talked about it.

Mostly. Yes.

Yeah.

Yeah.

So these. This was.

Yeah.

Okay. And this matter. Information.

Yes.

Carlo.

Yeah.

So I think, you know, if we have a rich schema, I think it would be useful to have this kind of stuff. And we were talking about this with Rob because Rob had some, like, he kind of faked this by saying, like.

What is this for? I don't understand what this is for.

Okay, let's say you have a hundred columns.

Yep.

And, you know, you probably don't care about all of them. You want to start exploring what's in there and you say, oh, well, let's see. All of those that, you know, don't support missings or those. All those who have a certain name or a certain type.

What is the thing for. Wait a minute. Column. What's the name for? I just want to construct. I just want to get certain columns. How do you do that?

Oh, you do tabular of all comma column names.

I use part.

You can use part as well, right?

You can use part, and you can use also column construct. Column construct.

That's what I was looking for.

Yeah.

I thought it was construct columns.

Yeah, construct columns.

Okay. And then I have column One column, two, et cetera, et cetera, et cetera.

Yeah, so what I was saying is that in some previous version, Rob was compiling things like all true integer Q.

What is this for? Are you talking about you're manipulating the list of columns in some way or what are you talking about?

No, I'm manipulating the list of names of columns.

Here's a concrete example.

How do I extract the list of names of columns?

Like I think tab.

Column keys.

Column keys?

Yeah, there's a function column keys.

But there's also a string sub value, right? Yes.

Okay. So I've got the column keys and I want to extract certain columns. So I say column keys, I do my selections and I feed it into construct columns.

Yeah, but the point is that, you know, the right way to do that would be a select. Well, I'm missing a keys there. You do a select on the schema, not on the data itself.

I know, but. So why are we not just doing construct columns of tabular comma, select of column keys, blah, blah, blah.

Because I mean, select of column keys only operates on the strings, cannot operate on anything else.

So what else would you like it to operate on?

On the type on, you know, things like whether this is a column key, whether, you know, it has a certain default value.

You know it's a column key. You just extract it as a column key.

Yeah, but, but here's a, here's a concrete.

Let me.

Can I give a concrete example?

Okay. Can I understand if this is, if this, these are. This is metadata for tabular column, correct? You're talking about the metadata for tabular column.

Yes.

Right. So, I mean, this is really.

No, no, wait.

Sorry.

Sorry.

Yes.

It's not metadata about the tabular column. It's about metadata about the whole tabular. Because things like key columns are not properties of the column or properties of the tabular.

Yeah.

Okay. Rob, why don't you give your example?

Yeah, my, my example is. Okay, and, and this is. So suppose you, suppose you have a big table with, you know, 50 columns or something, and you want to. And you want to. And you want to take the mean of the columns that are numerical or floating point. Okay, so you want us, you want it. Some easy way to select which columns you want to take the mean of. So you can instruct the functions for those. Yes, that's, that's. That. I think that's pretty.

Where is that data.

That'S stored? That's stored in the, the type. The name of the column is stored in the tabular, but the actual Type is stored in the tabular column.

Okay, well what's the minimal code to, to extract this should be.

It's like, is it like schema, like the way I typed it?

I don't know. But what's the minimal code if you're just doing it without any special functions to do it? How do you do it?

Tabular column of column, of quote column types.

No, wait, it's not of tab.

It's, no, not tabular. I mean tabular of quote column types. And that gives an association with the type.

Okay, so this will be column types here.

Y.

Correct.

Right.

Okay, so then we have it. Let, let's move on. Let's try and make some progress onto some other things. This, this seems to me like not something we need a special function for understanding. Understanding what's going to happen is, is, you know, it's complicated enough without having an additional function that you have to learn.

Okay.

I mean I, I don't, I just don't see it. Okay, let's talk about pivoting here for a minute.

Well, we're not, yeah.

Are we not ready for this?

We're not really ready for the pivoting stuff. I mean if you want to look at the ref pages, I think, I think there are examples there that, that you could look at.

Okay. But I'm just going to get a sense roadmap here of the full set of things we still have to review for tabular.

Well, one big part that we haven't started yet, but we'll probably be able to start soonish. So I mean the build, the last build failed so I, I, I couldn't prepare anything with it. But Rob started putting in the hooks for us to plug in directly into the same symbol, the out of course stuff. So we'll have to review some of the out of core stuff because there are some dispensations to be made about like what is the constructor, what the full form looks like, how eager or lazy do we want to be with errors? Like do we want to have the compilation errors right away or do we want to wait until the person says.

Yeah, okay, there's no need to ask the questions right now. I mean. Yeah, right, we understand there is some review of that.

Then pivoting. Well, we have to remember that this is like some sort of, of a smaller version of pivoting. It's not the full labeled array thing. So we need to decide what is useful here. What keep.

Yep.

Before having like a more full fledged object that supports.

Yes.

More cooler stuff.

Okay. And what else? I mean, we'd looked at a bunch of visualization things. Have we finished that review?

No, I don't think so. So the best thing there is a guide page called tabular processing New, I think that has the projections.

But now let's see who. Who's project managing here?

Me, Me.

Katla, over here. Okay, good. So can you make sure that you have for the next meeting a checklist of all the areas that we are supposed to be reviewing for tabular and the status? Yes, well, that depends on, you know, people getting back to her. Well, that's her job, is to get people to get back to her. Yeah, but you know, Roger.

Well, I think there is another thing.

The correct spelling. Is that the correct spelling?

No, no, no, no. Lowercase eu.

Lear.

I mean. I mean, Sean is not here now, but I think it's.

Sorry, it's a futurized, rough page.

I know. I. I went to this. Oh, no, it's. It's a guide page.

Yeah, guide page, sorry.

Yes.

So if I recall correctly, the decisions we have made about the.

What's it called?

The object Data connection. Object was also predicated on the fact that we were going to put this report in one. So maybe we should revisit some of those decisions now that we have more time.

Yes. So I think this is the fundamental picture with the areas. And so we have been focusing mostly on the object and cleaning and transformation.

Okay. So we look at visualization and we got to look at modeling and of course there's typesetting. Indeed. And what's the status of that?

Well, let's see. So Jason Abernathy has been working on that and we have been working on special processing of individual types of columns, particularly number numbers, dates, quantities.

Yeah, it's got to be. It's got to be strictly better than data set.

Yes, I think it's already better because it aligns numbers, it takes precisions into account.

Okay. Okay. And then we'll need to reverse, you know, we'll need to port that back into data set. Okay. Okay. What is publishing? What is that?

I don't know. This, this. I think this is the last section. This is Rogers and he was thinking probably on how to.

I don't know.

Make web pages out of tabular, things like that.

Well, that would probably be type, but go ahead. Well, you can click on it and we can see if there's. If there's something there to look at. Can you increase the magnification? I can't read that. Yeah, this is pretty aspirational. This is not the right way to do this. Not at all, not at all, not at all.

Oh, by the way, in terms of, in terms of transformation, before I forget, do you ever want to discuss window functions, not necessarily just within tabular, but also within. I don't know.

Ild you mean things that like map with a windowing function?

Yes.

We can discuss it along with supplied.

Again, discuss it along with supplied.

I think it's a reasonable thing to discuss, but we need to discuss supplied as well.

If you open this point, this topic, then I have to say that for the transformation language, we didn't like, there are a lot more things that we haven't been discussing at all so far. Like for example, for databases we used subqueries which make the language much more powerful and there are many, many other things like relations, etc. We didn't mention those because they are more advanced, but they're very powerful.

Yeah.

We'Ll show you, we'll show you when we get there.

Right. I'm not, I mean, I'm not convinced about windowing. I am fairly convinced about supplied.

Well, the thing about windowing is that it's a little bit difficult to explain to people, but like, at least internally, once you grok it, you can write all the things like accumulate, et cetera. In terms of that case of that.

Yeah, you know what? For windowing, I might have some way to express it in primitives because I did some experiments on that to express those in primitives, which will be much more understandable for orphan language users.

Okay. But you know, there are things about what happens at the end of the list, et cetera, et cetera, et cetera, et cetera, et cetera. There's all kinds of hair.

Well, we already have, remember, we already have a thing, I think it's called partition map. I can't remember if it had a name change. Maybe it had a name change somewhere that, that sort of does a multi d version of, of a moving type.

Block map. It's called block map.

That's what.

Okay, yeah, but the problem is that that kind of thing is difficult to use within transform columns, even accumulate is something that doesn't quite. I agree. It's a different one things.

So.

Yeah, no, it is different, but we do have something, you know, to, to look at and compare to.

All right, Okay. Is there anything else we really need to cover today?

No, I don't think so.

Okay, great. Well, we're making progress, but we need to keep pushing because, you know, this is right after the summer. We need to be wrapping this up. Okay, Thanks a lot. See you all soon.