Live CEOing Ep 815: Language Design in Wolfram Language [Tabular]

Hello everyone. Welcome to another Wolfram language design review about tabular. Okay, do we want to look at the recap first or do we not need to?

Well, the, the initial part of today's agenda is sort of a recap already, so I would skip that.

Okay. All right.

Okay.

Rename car.

So these are functions that we went through already and this is just to make sure that we are happy with design and we can like close it and, and move on. I mean the, the today is like data cleaning. So we have a question.

When we have the, the custom named things, do we know the syntax for a custom name column here? What is it called? Is it the multi argument key or how does it work?

It was like extended key, prefix key. I don't remember. What was the final decision? Rob, do you remember what was the final one?

It's extended key.

Extended key.

Okay, so that's what I'd write in here. If I want deleted. If I want to delete an extended key column, that's what I would write here.

Yes, exactly.

Okay.

Okay. So what does this do? What does insert columns do if the column does not have the right number of elements for the number of rows.

Either padding or failing, depending on failure mode, I guess. How do you specify by default we could paddle. The discussion about how to fail is a separate discussion that you want to have around every tabular operation. Because this is something that's, that's going to happen in a lot of cases. Yeah, sorry.

So I mean even in delete columns there could be a partial failure, as in one of the columns is not deletable, but the other ones are. What do we do in that case? You understand what I'm saying? A is there and can be deleted, but B doesn't exist. Do we delete?

There are arguments for both. Like, you know, if I want to delete A and B and that's not possible because one of the two doesn't exist. But that's not possible. I might want to be informed and reconsider. Maybe I did something wrong in my code. So there are always.

What would happen with a function? Like what would delete cases do? It would just not. Well, delete cases is a matchup, not delete. And it would not delete it. And it would just delete what?

It could delete key drop and key take. They ignore stuff that doesn't match. So if you have key take on association and there are keys ABC and you only have a B, it will take A and B, it will do intersection. So we can, I mean we discussed many times we can be lenient in the default case. It would be useful for pipelines to have a way to maybe.

I mean, that feels a lot like some of these failure things, like the whole confirm, you know, whatever thing. I mean, you know, maybe there should be a wrapper that is a, you know, controlled evaluation wrapper that says something like, you know, strict operations or something. I mean, that might be a useful thing in general.

I mean, whatever we do. Like, I think that, for example, if you take delete, delete, you pass a string with a single element. Sorry, a list with a single element, and you try to delete the second that issues a message and returns unevaluated. I think that would be the worst case. Like we either return failure or return the tabular with undeleted stuff. Those are the only possible tool modes, I think.

Well, okay, anybody? I mean, this is the thing we're going to have to discuss in a subsequent meeting. But the idea of a wrapper that is a strict block type thing, Is it crazy or not?

I don't know. How do you. I'm not familiar with enough with the evolution semantic. Like, is it a headdresser for the function or. It is.

No, I was thinking it was a wrapper. I was thinking what it would be.

So it's like a block construct evaluate.

Of, you know, delete columns or something. I mean, maybe. This is crazy.

Yeah.

I mean, it feels a little bit like. I think it's a little crazy. I mean, I think it feels a little bit like, like. Well, I know parallelize actually only goes one layer in, but. But it's a little bit like something like parallelize where. Yeah, it just becomes very imprecise and it's not. Yeah. You don't trust it to, you know. Yeah, yeah.

Well, it also has to be hold first or hold all. And people always have headaches with hold stuff.

Well, is a headdress more interesting?

Why not an option? Yeah, I guess that seems like that's the way we would do this anywhere else.

Okay, fine.

I would do an option with a dollar variable. Yeah, it's a default value for the option. So you can set it for your whole session and be happy if you want. And if you don't care about it, you will never see it.

Or if you do want a wrapper, you just use block.

Yeah, right.

Exactly. Yeah.

So what would it be? It would be something like.

Well, I think it's sort of like the fade interaction we talked about.

Yeah, yeah, yeah.

And then that could either be a very simple thing or it could be a whole association of different kinds of things to do.

I was thinking association. Yeah. If we have more, that gives flexibility. If we have more things to control.

Okay. All right. Combine.

Sorry. Just, just, just, just before we move, are we fine with the first three?

Yes.

Okay, great. So the second thing that means insert column is basically anything that can be converted to tabular column, right?

That would be the case with unknown padding, which I think could be the third argument.

Do we support lists for insertion?

I would hope so. This one here?

No, no, no. I mean, I insert A. Can I insert A and column B? Because if we do that, padding doesn't make sense. Or it has to be a list of the same length or an association.

No, I'm saying that here.

Yes.

I would assume that we can say, you know, A goes to this, you know, B goes to whatever. B goes to something.

Exactly.

But that could be an association as well.

But my, my point is that for typed something like this, which is not a matrix, is like different types, the padding needs to be specific by column.

I see what you're saying. So I mean, probably that has to be then maybe a third argument, which is. Well, okay, so there's two possibilities here. Either the third argument, which I don't like very much, is the padding argument, which says what to do, or perhaps better. And this is an interesting construct in general, one could say padded of this comma, blah, blah, blah. And that would be an interesting construct.

And I, I, I, I love that you are creating wrappers faster that you can even the line. Right.

But in my opinion, I think this insert columns is nice as it is because it's very simple. If we want maybe, maybe an easier way to do padding is just to set up pad, you know, padding for tabular column and just have that work. And then they could put the padding they want, you know, padded or padded, you know, or pad left or pad right around the tabular column thing.

That's possible. I think this is a somewhat interesting construct in general that I can imagine being useful.

How is that connected with the ray pad is like an inactivate.

No, I feel like it would be.

Some kind of not inactivate. It's more like an operator form.

You don't have thing that does an operation.

Exactly. This is a pad, right? Where do you provide later the length you want to part to?

I think so. I mean, it would be something where you'd say padded of this, you know, for example, let's say you say partition padded this may not be a good idea. Okay. It may not be a good idea, You know, like this. And it would do the obvious thing. It may not be a good idea. But I like.

It's complicated because you would like extra arguments left. Right.

With the extra arguments of partition. That would be a complete nightmare.

Exactly what's about to go there.

Yeah. By the way, I hope we have an LCC discussion about an association form of partition that gives us named. A named way of accessing its. Its various features.

I mean. Well, I. I don't think that's.

I don't.

We can talk about it. I mean, we've done some looking at it. I'm not sure that's really going to solve the problem, but.

Okay.

All right.

Can we make sure that that's brought into an lcc?

It's on the list even already.

Okay, good, good, good. Okay, are we ready to go on here?

Yes. So the design for this is something that we looked at, but more quickly than the stuff above. So this needs a final approval. Now we have two functions, combine columns and separate columns. The proposal here was to substitute them in transform columns. Combine columns is fairly simple in my opinion. So the old syntax was like A and B gets combined in C using the function F. And you can rewrite that fairly quickly using transform columns.

I consider this fairly obscure.

Yeah.

Right.

But what you can't do is to have simple idioms, string names of common cases in the transform columns.

Yeah, that was the advantage of combined columns.

What do you want as the idioms?

Well, I mean, look at the. Look at the reference page. Well, it's called column combined. It was just a rename there in the reference page.

Column combined.

Yeah.

Well, I don't see anything special here.

Common combiner function cfunc. Okay, but these are just functions. I think the one that was more interesting was column separate. It was taking special things.

Yeah, I think maybe we didn't work on this one in that sense.

Right. Okay, so let's understand these things. Okay, so separate columns, column C.

Okay, so the extension for separate column is slightly more convoluted in a sense. That is an extension of the syntax of transform columns where you allow the left hand side to be a list. Exactly. And here there's also again, we go back to failure action, because this of course makes sense only if F is producing F applied on column C is producing a list of two elements. Now, again, you have multiple things you can do, you can pay, you can do. Missing. And I guess these two are the one that. That are Most would be.

Okay. So let's just try to understand this, okay. This thing where we've got these two columns and. I mean, I'm not convinced that we need anything special here because I feel like, I mean, don't we have the notion of a date value, for example, that will be. Exactly. Let me just write out what it would be. Okay, so if I say date value of now, comma, our minute. Beautiful. See what I'm saying? So now the question is, I don't know whether date value has a. Has an operator form, for example. I don't think it's particularly necessary. I mean, in other words, the. You got to say what column is going in anyway. So, I mean, to me, that's pretty elegant. See what I'm saying?

Yeah.

And it also solves the problem with the design notebook that why should Dave, date value or date give our, you know, year, month, day versus some other list?

Yeah, I mean, I think this is the right way to do it. I don't think. I think you can't satisfy people. You know, it just. I don't. I don't see a. This seems pretty nice to me. Now, this particular thing here where you're saying, spread this into a list. I don't know. You'd have to know how to do it. But once you know it, it's. I think it's going to be fairly natural and I think it's. And presumably you can give a list of things here where it can either be, you know, a single column is combining or two columns are being. I think this looks pretty good to me.

Yeah, yeah, yeah. You can do several operations together and also this work in transform columns. And of course, if it works in transform columns, it will work in construct columns. So if you want to do on, you know, only those two columns, that is an advantage.

More than that, construct columns. Oh, I see what you're saying. Yes, yes.

Is the one that makes only the columns that you list. Excuse me.

Right, right. I like it. I like it. And I like the fact that you can, you know, separate, combine, etc. In a big list of things. Question from Seth on our live stream, insert and insert. The third argument specifies the location. Will insert columns have a similar feature? Well, that's relating to this padding stuff and so on as well.

But remember, Stephen, it uses your anchor column idea. Do you remember what that is?

No, I remember what that is. But. But I'm. There's two different issues and I don't know quite what Seth is referring to here. Actually, one is where is the new column inserted? And the other is, I think.

Is that. I think. Is that because. Okay, because the last argument. Remind me how the.

I mean, you say column and then it's. It's inserted after that column. Right?

Right.

Yes, exactly. So once. So you could just use a. So say. Say it had column, you know, D. You could put D, comma, a goes to pattern, and A would be inserted immediately after D.

Right. So. Okay, so let's. Let's walk through that for a second. So it is things without rules.

Yep.

Our anchors. So, in fact, presumably there's a. I mean, I think it's actually fairly nice that. That.

Only downside I can see of it is if. If you want your column to go at the very front of everything, it's kind of hard to do that.

Why. Why don't you just do this? Why don't you put. I mean, and this should work. If X is the first column, then just put it there. You know, it doesn't have to be behind. It has to be. Look, what has to be true is it's a bunch of triple blankery. Okay? It's basically the. The first element in the match from a bunch of triple blankery. Am I making sense? If you take the column names and you go triple blank X comma. I don't know if this is. You get what I'm saying? You're finding.

No, I get what you're saying. But if X is not included, we still put. We append at the end, right?

That's correct.

Yeah.

But X is an anchor, so it says you've got to have it be in the order where this. Where X comes after this column. After the A column.

Yeah. So one question here. What if your original tabular has XY and you write here Y, X, like Y A comma. Okay? So we don't allow to reorder things in this. With this syntax. Okay.

The only problem I can see with that case is if you do say A goes to. So you just rule A with rule with a, rule with B. And then you use A as an anchor column.

You can't do that. How can you insert a column? You've got a column called A. So A is not an. You know, A can't be an anchor. I mean, sorry, you're inserting a column called A, and there can't be two columns both called A. So therefore, you know, A can't already be present. So therefore it can't be an anchor column.

Ah, but see, the thing is that's very. So transform Columns. The way it works as it goes through the successive things is with fold. So every time you put in a new column, you can use that new column in subsequent rules in the functions.

Wow. I would never have guessed that.

Yeah, that's.

I would never have guessed that.

Well, I think it's.

I predict there's going to be scoping trouble with that idea. I am certain there's going to be scope.

I think I look at it once again.

Database does that. Does stuff does that as well.

So say it again. What?

Database stuff does that as well.

Well, that's not an argument for it. I think that's weird, to be honest. I think, you know, hopefully nobody will ever use it, but I think it's weird to not be, to be sort of stateful. Like, Let's, let's. I, I feel that this anchor thing is pretty nice and I, I think, I don't see a fundamental problem with it. And the idea that you can use a. And then use it again as an anchor seems crazy to me. If you that concerned about how it's being set up, use a different mechanism.

Yeah, fine, fine. We won't, we won't allow a. If A is created as a column, we won't allow it to be used as an anchor.

No, I mean rename columns. Can that be done to reorder columns or do we have a separate reorder columns?

Well, you don't need that. You can use transform columns to reorder the columns.

Rename columns. Rename columns has a list argument. That's not rules, it's just names. It has to match the length of the number of columns. And then that's a, that, that's a reordering.

But isn't it obscure to do it through rename?

Yeah, I think it's obscure. I mean, I think if we're, I mean, there's a certain argument that we should have a reorder columns.

I don't remember. Rob, do we. Do we allow naked column names in construct columns and transform columns? Because if we do, you can't do.

It in transform columns you couldn't do that because you've got to have anchors there. See what I'm saying?

Yeah, but in construct columns you can.

Do that, but it's not good.

You can say. And also with reorder column you have the same problem. Are you reordering partially reordering or you're creating a new one with a new order?

I'm sorry, with reorder columns, we had.

The reorder columns and we rejected It. Because it's just using construct columns, right?

Yeah.

If you're literally. Unless we have some syntax to only somehow specify only a subset of the columns, it's just construct columns with, you know, some permutation of the existing columns.

But how do you actually do it with construct columns?

I mean, I think that in construct columns, if you. We can.

Don't you have to say A goes to hash a?

No, no.

A can. A can appear by itself and construct columns to mean you want to include that column at that position in the constructed thing.

I see, so. So Dylan doesn't need a function after.

Right.

So it's a little weird that in one case it's an anchoring. Well, actually, it's not so weird, I guess. Right, okay. So in that case it is the sorting, and in the transform case, it's anchoring. That makes sense.

Yeah. The thing that really bothers me about this is that I think that it's going to be quite common that one does transform columns. Transform columns. Transform columns a bunch of times. And at the end, for presentation sake, or for bringing in just a subset of the data, one is going to have to do construct columns to just select a subset of the columns.

Yes. What's wrong with that?

There's no construct. Like, if you do.

Wait, wait. Sorry. Sorry, I'm stupid. You can do that with part as well? Yeah, of course. Yes, but part, why do you need. Oh, yeah.

So I think reorder columns is not a good function to have unless we have something that reorders a column based on some criteria in the column, which I think like is not a good idea.

Yeah, yeah.

What I was saying is that in general, the problem part is that part cannot have like. Since it's vararg, you cannot do an operator form of part. So if you're doing like, since Seth is listening to us, if you're doing Seth style, slash, slash, kind of chaining, there's no way to do that part. So you're forced to use construct columns.

Yeah. And what's the problem with that?

I don't like. I mean, if we're talking about having functions aliases to do things, I don't like the fact that there is no like construct columns doesn't explicitly state what you're doing there if you just have strings.

Well, what do you mean? You're saying that it doesn't. It doesn't say it's reordering. Look, I think.

I don't care about reordering, Steven. I'm saying you do a long pipeline of work. And in the end you care only about the last three columns. And so you want to say just, just display A, B and C. The only way to do that in a pipeline is with construct columns. And I think it's. It's not the name I would grasp for to do that operation.

So it's not like a self documenting function call.

Yeah, I mean, look, I understand your objection. I think we should go ahead with this right now. If it turns out that we are doing this all the time and people find it obscure, we can have a reorder columns. The functionality of construct columns will be there. Let's see if people are okay with it. Make sense? Yeah, because to me this is not that horrifying. But. Okay, let's keep going. All right, we wanted data cleaning and this is. This is about missing right here. And are we ready to talk about pivoting or not?

I would talk about mixing today, not pivoting.

Okay. What is pivoting?

Position is why this doesn't. Are you on. Not on prototype?

I. No, I'm not on the prototype build.

Ah, okay. Okay. Okay, fine. Then this is reason why.

Okay. All right, let's go back to missing here.

Okay, so here there's some. An inventory of things that, you know, the point is like what. What can. What can be encoded with the missing thing we have. Yeah.

What. Which of these is. Does confirm deal with Confirm do.

Good point, good point. We should enclose like composition confirm on objects and figure it out so it's.

It's a different set first. There's some here that maybe are. I think confirm handles some things like maybe like dollar canceled or something like that I should check. I think it's documented though. But there's several of these that confirm doesn't handle. Like, I don't think Confirm handles indeterminate or undefined. Overflow, underflow.

Maybe it should.

Yeah, yeah, Confirm. It's in the confirm reference page is the list.

Oh my. I don't have in product documentation for some reason didn't install. So who knows what this is? Okay, fine. Okay, so this is an interesting inventory there for confirm. Probably should be extended to those cases.

Okay, well there was some like indeterminate. Where I'm not sure. Confirm should fail on indeterminate.

Fair point.

The one I would really care about because it happened to me is terminated evaluation. Yeah, that.

That post dates confirmed. So yeah, I think.

No, no, I know, I know. But, but, but yeah, this is really something that I don't. You know, that's something right there is a failure and I.

There's no reason for it to not handle it. So.

Yeah, exactly like. It's not exactly what they wanted.

I really hope that time series gets finished soon. It's really upsetting to me that that's not done yet.

Christopher, is this parameterizable for confirm? Like is what for confirming what if I can. If I care about failures, but I don't want to consider missing failures.

Then you should use Confirm buy or Confirm match. I mean, confirm is basically just a special case of confirmed by or confirmed match with sort of a hopefully useful default.

Okay. All right, let's take a look here. Okay. Missing data method. That one missing behavior option. Gosh.

And Jose did a good job in updating the documentation of missing behavior. You asked for it. So if you want to open the ref page in cds, there are new examples that explain in detail. Because missing behavior has two values, none or automatic. None is trivial. It doesn't do anything with that. But the automatic one is doing several things depending on the category of function that that it applied. And I think it's very interesting for what we want to do in tabular. Jose, you can confirm or deny that this is an updated page.

Yeah, I added scope examples trying to explain those different categories. Yes. So I think the idea here is what to do when the function act acts on a list of values and one is a missing. The automatic value is the automatic behavior is to ignore that missing. Then there are other cases in which we have vararch functions, other cases in which we have empty lists. And so rather than a behavior of what to do with a missing, it's when to return a missing.

What do you mean when to return? Okay, let's get some scope here for.

What we're trying to do.

So.

So the. These are the current options we have to control missing behavior which have been introduced in different versions with different needs. So the right is focusing. The right notebook is focusing on missing behavior.

Right. Okay. So let's just review what these things are. So missing data method is something that interpolates missing data. Is that a true statement? Roger?

It can interpolate. Yes, but it also replace with some fixed value.

Okay, fine. All right. Missing behavior does what?

So it has two values, automatic and none. Automatic means for each individual function it will try to do something that is not nice to that function. Sometimes it's ignoring the presence of missing, sometimes it's producing a missing.

Yeah.

So for example, if I say one plus missing, what does it do? The missing behavior to automatic.

With plus it will ignore that missing was there.

So it'll be just one. It'll be as if it's zero.

Exactly. But with times it will return missing.

For example.

So the behavior is specific to each function and I have those lists in those excluded cells, you see the list of functions that are prepared to do that particular thing.

So in general, what it's saying is if there's a missing in the picture, what is it doing? It's saying if there's a missing somewhere in the function. Gosh, let's see. I mean, it's as if there's another function that is invoked if there is a missing in the function. Because in these things like plus and times, their own evaluation functions are invoked. And it's only because the missing just sits there symbolically that this missing cleaner ever comes through.

Exactly.

Yep.

Okay, So missing behavior, just, just so I make a note here, only relevant for functions like plus and times, for functions that actually will actually return a symbolic missing.

Well, it's so important for all the statistics functions too.

And those act on list because mean, I mean, if I say mean of 1, 2 missing, I get back a missing there.

Exactly. Now wrap the mean in head with query.

What do I do?

Just another bracket.

Yep.

So they are missing was ignored.

This is completely ignored. Right in the denominator as well.

Yeah.

Just to add a data point to this, in general, what happens.

As far.

As I know in data frames and SQL is that things like mean behave generally differently from plus and times and so on. So they're considered aggregations. And for aggregations missings are ignored, and for row by row operation, they're contagious, like indeterminate. And so I think this is why we have a different behavior for times and plus in this case is that plus behaves like an aggregator and times doesn't.

Well, also, just the meaning of not including missing would be a little bit different because plus, I guess, well, I guess that doesn't apply to this year. But I mean, it's just the identity would be different. I could be a zero for plus and one for times.

Right.

But the stickiness of missing goes far beyond that because even Boolean operations can return missing. So if you compare, the question is like is one equal to missing? You can either say false or you can say missing.

Well, missing is a weird thing to say for that. What happens with this? Does anything do something with this?

I don't think so.

What do you mean?

Well, if I do this, for example.

I mean, triple equals will certainly say no.

Yeah, no, I don't Think equal is supported in query especially.

Okay.

I don't know why actually given this we have. Maybe we don't have logical operations in general like greater less.

But in a.

Certain sense, I don't know.

Sorry. The Wolfram language is well prepared for like 3 valued logic here. If you think about like 4 argument if and the fact that for example comparisons between things that are not well ordered return unevaluated or even equal can return unevaluated. That is kind of three valued logic, right?

Yes.

Okay, let's. Missing data rules. What does this do? Semantic import.

This is only used for semantic import.

What should be considered missing.

Yeah, this is.

Oh, okay, that's on data import and that's saying which is something relevant to us. It should be. That should be missing data rules should be added to things like the CSV importer, should they not?

Well, I think if you scroll down the synthesized missing values has an option which is called missing value pattern and we can look at that later. But this is closed machine learning. Missing value synthesis and missing value pattern. But the point is, given that we have an option which is called missing value pattern, we have a long standing tradition of having patterns followed by rule to do replacement. This way you could do that. Missing value pattern is what you consider missing in the subsequent computation.

Because you're proposing. This is crazy. Oh, missing value pattern is that. I see. So that's what would be considered missing. But that's not saying what to do with it. Let's just say exactly is missing. But my point is could turn it into missing of something or other.

Yeah, but my, my point is that pattern and I don't know, maybe you can kill me here, but in replace you have a pattern followed by a rule. So I don't see anything crazy in having missing value pattern as a list of underscore integer or like I underscore integer goes to missing. I.

No, I understand. I mean this is what you're saying is what is missing transformation.

This is another topic. Another topic.

Okay, all right. Okay. So the first question is. Okay, there's two questions in the importer. How do you go from a random thing like NA to a symbolic missing. That's question one. Agreed.

Also is shouldn't that. Is that supported by import or we put there as something that we would like to be supported by import. It.

It's not. At least not in CSV.

I'm not aware. Right.

Okay. And I don't know why we had.

I don't think it's got it anywhere. But I think we should have It. I think it's a very useful thing to have.

Is there always a, a pattern like, like, like that? Like, I don't know if you're importing from a. Trying to think of a harder case. Like if you're importing from a shape file or something, so you have some geometry and then some data associated with it. Would there be some analogous thing? You might want to be able to.

Say.

I don't know what it comes in as. I mean, yes, you might have, you.

Might say mental models.

Look, look, here's, here's what you really want. I think the missing value pattern applies to the thing that is brought in by the importer, the raw thing brought in by the importer. It's essentially a post processor on. It's essentially, you know, what it essentially is is you know, replace with missing, blah blah, blah, blah blah. And then, you know, na or whatever else.

A few, a few questions. So the first thing is. Okay, a couple thoughts. So the first thing is that this could be a format specific option. Like we have a lot of options. Like, you know, there are options that only work with CSV.

Sure.

You know, and I would argue maybe this should be something like that because my guess would be that one, a lot of different options have different sorts of explaining what this pattern is even going to be matched to is going to be format specific. Like is it matching to rows or is it matching to elements or is it matching, you know, and JSON, what is it matching to, you know, etc. The other thing would be there are probably efficiency things that are again format specific. So for example, when we import a CSV, maybe PIOTR could say more about this. First you can get it as just a bunch of strings and then you can get it as, as numbers and things like strings and numbers, depending on if your numeric is true. Right. And so for example, would you want this to apply before or after? Because it might be inefficient if it does it after.

Right.

I think it's best to apply it during parsing of the file. That would be the most efficient and that we know that it actually is possible to create tabular or in a sense it's possible to. After that replacement, it's possible to import this as tabular column.

Okay, so in that case though, so I mean is that basically you're using, you're using some library to do CSV import and it supports some sort of syntax where you can tell it to.

Right.

So arrow library calls it null values. So we can basically use that Option of the arrow library and pass this info.

But.

So for example, the answer is it depends.

Right. Which is, which is why I was arguing maybe there should be a format specific option that we can then write, you know, a paragraph about for CSV and another paragraph about for JSON.

Well, but. Okay, so. But it could still be called missing value pattern, but its interpretation could be format dependent.

So what about, what about a situation where maybe. Well, suppose that we have a similar thing where you can make an entire row missing or something like that. Or like ignore entire rows. Maybe. Maybe that would be a different option.

Like.

I don't know. I'm just trying to see if there are cases where.

Yes.

Where you might want to have multiple options to do this for some formats.

Well, okay, ignore a row. What other operation do we have? Let's say a row is full of missings. What other way do we have to drop that row?

Oh, oh wait. Actually I, I have a, I have. Sorry. Here's a case where you wouldn't want, or you'd at least want something a little more complicated would be. In this column, A value of 65536 is considered null. But in this other column it's. That would be real data.

Yep, yep.

I mean, that's fine. You could have a syntax for that. But again, it would be very format specific. Probably whatever syntax you'd have for that would have to be different for JSON.

Right.

But then even within CSV, how you would want to parse it would change depending on what you wanted to match for certain patterns. If you know ahead of time that you want this particular value to match like that. It's very different if you have 10 gigabytes of data that you're parsing. So you have to do some sort of inspection first to figure out which patterns you would even want to represent as missing with large data. Wait a second, that's where the parser concerns come in.

Well, wait, you're telling us.

Here's the suggestion. First point is, what about an import element for CSV that looks for weird stuff, that looks for non numeric stuff, for example, and just returns, you know, the list of non numeric things. Or for example, you know, it could be non numeric counts or something. I don't know if that's the right, right term, but that would be. That would give you an association of the counts of, you know, this number of nas, this number of whatever else. I mean, that would be useful in the 10 gigabyte case, right? That would be a highly useful thing to be able to get.

It's only quick to parse that to get that information if it's malforming in the 10 gigabyte case though, problem. So what if it's well formed, well formed data, right? Because if you are parsing it and.

Well, I'm sure you can find an efficient way to parse to just search for the weird things in CSV. I mean, you know, it's a regex basically, right?

And the point is essentially that going into the import ahead of time, which backend you use to get which piece of information will. Will change a certain back end.

You're not understanding what I'm saying. What I'm saying is, I'm saying you say import, you know, file CSV, okay? And then comma, non numeric counts. That's all you're doing. That's all you're trying to get from this. You're not trying to actually get values. You're just trying to do a probe here. See what I'm saying? And that probe can be blindingly fast. Done with a, done with reg exes and things.

Well it can't be. I mean it's supposed to go through the whole CSV.

It goes through it as fast as it can get through 10 gigabytes essentially. But that's why I'm saying if you have some information about what the file is, you can be smarter about how you're passing it and do it lightning fast. So the more assumptions you make going into parsing ahead of time, the faster.

You can do it.

If you want to check for those more flexible things, you probably want to take a look at a smaller chunk of data.

Actually I noticed ahoy on our livestream commenting that they often end up doing the, you know, in pieces. What amounts to this operation, this operation is clearly useful for very large files. It's clearly a probe that you want to have.

Well, but you know, I assume you, you'd probably have to give it. So first of all, I'm not sure it's non numeric because CSPs all kind of strings. It's really just stuff that doesn't match. Like you'd want to run import and give it some kind of basically column type signature or I guess it might be can derive one or something like that. I don't know. But like it would be how many things was there a value there that had to be discarded because it didn't match the types that were expected?

That's what you probably care about, anomaly counts or something. But I mean again that that could be. Look, the fact is, in a perfect world, you know, you'd want anomaly counts. Anomaly counts by column. The first thing is just the lazy version of what's the weird stuff in here? And the second version is per column. What's the weird stuff?

Is there a reason to expect that this would be. I mean.

Why.

So, okay, I guess there's some memory efficiency reasons, but why would you want to do this? As opposed to just try importing it as a tabular and then counting the number of missing in each column or something like that?

Because it's a horrifying memory use. You know, if it's 10 gigabytes, okay, and it turns out there's one weird missing somewhere, it'll be super useful to know that, I'm sure, for Sean's, you know, blinding fast CSV import.

Aero is so fast and multi threaded that Even on the 10 gig case, it's really not that bad to try it first and see if it fails.

Well, whatever. That's an important.

That is a useful.

Right, but I mean, that's useful to know. But this is getting. This is a useful piece of information for any kind of data. It's like, how much stuff is there here? That's. That's sensible versus weird stuff.

But the question is almost first, like tabular Q. If I can import as a tabular.

Yes.

This isn't very important and we should just do that. If it isn't, then we should do a little bit more work like you're talking about.

Right. Unfortunately, I am going to have to go because I have to do something else here. We will have to continue this a different time. You. You guys could continue if you want, without me. So do you want to do that? That's a lot of unenthusiasm for that.

Sorry, I was muted. I mean, we reached the end of what we wanted to look at with you. We need to prepare material for people.

But I mean, you've already prepared something about missing. We just need more stuff with missing. I mean, there's missing importing and how you deal with missings. You know, how you deal with turning things into symbolic missings in the importer. And then there's all the stuff about how the Operations, the Fs and Transform columns and so on deal with missings.

Yes, yes.

Okay. To be continued another time. Sorry, I have to go. See you later.

Sa.