Live CEOing Ep 838: Language Design in Wolfram Language [Tabular]

Hello everyone. Welcome to yet another design review meeting about tabular. Hopefully nearing the end of the ones that we're doing for this coming version. Okay, what do we got today?

Okay, so we thought a bit about the design for missing handling has been basically handled last time and closed and we are all happy with that. Now anomalous data is slightly more problematic and we attempted the implementation and that with the ideas we had last time and that is highlighting like a few things that are unsolved. So I'm trying to be quick here. I put a summary of the current syntax of find anomalies, which is our current function to find anomalies and ideas to extend the design. So okay, yeah, I mean current version data data property or you can put.

Function right now is primarily anomaly detector.

Exactly. I put that below like a low distribution of anomaly detection.

A naked thing here.

Exactly. So we had here I all these examples are. Let's play with the design and see what can go wrong. Basically. So simple function not very problematic. Let's start with a univariate case where let's say we apply the detection on a tabular column on a single column. That's quite easy. So one limitation of this design is that the function ideally is mapped, which is fine for a lot of simple predicates. Like I want this value to be non negative or stuff like that. But it has the limitation that you cannot do stuff that requires the whole column usual thing. Now we have column wise value though, so we could work around that supporting column wise value here. Yeah, so what I have here in the example below, hash greater than colonized.

I mean that's if you really want to put it all into that one function. I mean you could also compute the mean and then do find anomalies relative to that.

Yeah, absolutely. But I'm just looking, you know, design limitation and how to circumvent that. Yeah, so yeah, this is not super problematic. So name methods also fine, you know like Z score or med or integral time range. All stuff that can be easily, easily applied on a single column. And this also works fine in transform column. Here I'm adding a weird column which is using column wise thread and basically giving the true false column a boolean column using this score for example on the column A. So this kind of works together in a sense.

Now anomaly boolean list is an existing property for it is it returns a.

List of booleans and this will be threaded across the whole column. Now multivariate case is a bit more problematic. So for the pure function case again Easy to do if you have a function. Yes.

I just want to say one thing. There's one problem is the fact that most of the third arguments of fine anomalies, on the other hand, do not return something that can be threaded back because it's either just the anomalies or just the non anomalous values. So in that respect you can get things that are difficult to play with.

Yeah, you can. It's a bit unfortunate that the default case here is not the mask, but is the list of anomalies. So you always have to force to get the mask out or do this Boolean list or something like that.

Maybe you have a function, a separate function that gives the mask, which is the primary thing used for tabular.

Maybe or maybe rather than the mask. The useful thing for tabular is. But we will get there with transform anomalies.

Just say find anomaly positions or something.

Find anomaly positions or that's going to be a list of positions.

Though the mask is not necessarily the most useful thing. The most useful thing is keep the data that is non anomalous and put missings instead of an anomalous or trim or clip.

Yeah, I mean those things. We have another function which is transform anomalies that can take care of in place modifying the column. But many of the problems for the multivariate case are absolutely the same. So I think starting from here. Exactly. So, okay, first example, I have a list of pairs, stuff like min or mean. If you map the function, again, no problem. We could even use the same semantic as select and just say that there's a true queue around that. So even the case Of # Greater than 2 is just not gonna hire. If you want to generalize to multivariate for separable conditions, you could use all true, any true, all true or that is it holds for every element, any true for at least one element. So that's also quite simple to exemplify with our existing stack. Now the problem here is with name methods. So name methods, they are confusing. How do you combine them? You.

Just. How many do we.

Have? I mean, three are the most common one which is one based on standard deviation, one based on interquartile range and one based on mean absolute.

Deviation. Okay, so what does Z score actually do? If Z score was a standalone function, what would it.

Do? Z score is just computing on a single column, is computing.

The distribution and then computing Z scores for everything based.

On. It's like the value minus the mean divided by standard deviation, absolute.

Value. So it's basically, I think when you're three standard deviations out or some other standard.

Factor. Exactly. So it's basically you do standardize of the column. You take absolute value and you check what's greater than.

Three. We don't have a thing called zscore list which takes a list and returns those values. I mean, just as a standalone.

Function.

Standardized. Standardized, yeah.

Standardized. It's not standardized. That's. It's not standardized. Is not a list of Z scores, is.

It? It's absolute. I mean, you need just to take the absolute value. Absolute value of standardized is the list of Z.

Scores. But also, also Stephen, even if it were a function, we just said that functions are mapped so it wouldn't work as a.

Function. You have to use column wise.

Thread. But Julio. Okay, Z score is clearly a scalar definition. Now there are different ways of getting multivariate definition. I think you should have something else like ellipsoids, you know, like.

The. Yeah, but Roger, one thing that is very.

Common. If it's multivariate, it damn well be.

Multivariate. Yeah, but Roger, one thing that is very, very common is that you have like 5 numeric columns and you want to consider any row that has at least one anomaly like invalid or you want to consider.

Invalid. Why is that common? I don't think that's common at.

All. It's a common cleaning thing. You don't want to do it. You know, it's not always useful to say this. And also the other point is that you don't want to call this function 15 times for 15 columns. So sometimes you want to map it across the columns. Sometimes you want to take the. And sometimes you want to take the.

Or. I see. Well then that's fine. But it's basically a bunch of univariate.

Methods. Also this is a different function. I mean, find anomalies is just finding them and returning them. Right. There's a different function which is essentially delete anomalies. Where did we land in the.

End? Handle them. I mean handle them. Whether you trim. Delete this Trim. Classically named and windsorise is kind of Map it into. To maybe those three standard deviations in sort of. Yeah, those. Those three sigmas out. I'm saying you projected into some.

Value. Yeah, exactly. Oh, I. Sorry below. I used stream for winds rising. Was my mistake, but yeah, yeah, exactly. These are typical things that you want to.

Do. Where did we land on the. What am I trying to say on the delete missing.

Stuff? I think the consensus was to use discard. You propose two new functions. I don't remember at the end of the discussion if you are still up for proposal or not. Like any missing, all missing discard.

Is implemented and documented.

Now. Exactly. Discard works very well. And we tried to design discard mixing. It was basically, it was just the same syntax as discard. So we just say let's not do.

That. Okay, so look, what the hell is the good of find anomalies typically? I mean transform anomalies I understand completely, but find anomalies I don't really think. I think it's a kind of a debugging type function, isn't it?

Yeah, I think that there's sort of a range of things that these things based on statistical distribution things are too far out in the tail flag it because that's probably something went wrong. Then there's something you have pre knowledge I know that this quantity cannot be negative. You know, if it's negative, it's definitely an anomaly type thing. You know, you have some outside knowledge that says where values need to.

Be. Right. But I mean and okay, so look in this thing about if you have different columns basically that you need to do different things with. I mean I can imagine a syntax which I think is not ambiguous of saying, you know, Z score comma. I mean maybe it's Z score comma, Z score comma blah blah blah, whatever another case is. And that means the different elements of the lists. And if it's, you know, if it's association thing, you could use that for, for named columns and so on or named association.

Elements. Okay, yeah, I mean this is verbose, but as a fallback, if you want to do what Carlo was saying to do multiple things in the same.

Fold. And the other thing is that if you want kind of a. A Z score for everything, I don't see why we shouldn't have a different name for that case. I mean, you know, maybe Z score, you know, I don't know whether Z score on its own would mean that doing the column wise value type thing or whether Z score on its own means something more specific, etc. Etc.

Etc. Yeah, this was something that Carlo suggested while we were meeting today. Like to have orthogonal names or stuff that is strictly.

Multivariate. I think that's a good.

Idea. So we can document that if just put that string it would work multivariate. Now conform anomalies, sorry, transform anomalies. Yeah, here by trim I meant what Roger called winsorizing. Basically when you fold back outliers within the distribution. So I mean the main point is how you want to Call it that this thing will return something of the same length with anomaly reduced back to non anomalous variant. So direct column application. This, this will take of course like a where and what to do argument. Because you need to, you need to say both you, you floated that syntax. Tabular goes to A if you want to select the column. No, if you implement this for like if you implement transform anomalies on a column. I'm not super excited of having transform missing work on tabular data and transform anomalies on column wise data, but maybe there's no way around.

That. Sorry, transform missing. Working on.

What? Transform missing takes in tabular data. It can handle a column, but cannot handle the whole tabular. And you specify what to do for each column here we didn't want the double rule on the second argument. You know, A goes to Z score goes to.

Train. Well, I think that might be the lesser of the.

Evils. Yeah, but also I'm thinking of why, if we have one that finds it, why is this one also finding.

It? You want to nest the function.

Call.

Yeah. Which one that finds.

It? I mean the, the previous one. Find anomalous. Finds.

Them. And this one, what do we do with them?

Right. This one does something to them. They transform them in the missing case. We've already found.

Them. Yeah, I understand, but what does find return. What does find anomalies return for a.

Tabular? I think what would be useful for it to return? The.

Mask. I mean it has properties, but by default. The default property right now of find anomaly is. Exactly. So it would be like a subtabular.

Yeah.

Right. So in, in this case, for transform anomalies, in the first argument, we don't want tabular, but we want something like find anomalies of tabular methods from a tabular and plug.

Tabular. Listen, if we want a function that returns the anomaly masked. I. I don't think it's the right thing. I just don't think it's the right thing. I think that unless we want something. No, I just don't think it's the right thing to have. We could have a separate function from find anomalies that produces an anomaly mask or alternatively a tabular with. With. What do they call? You know, with. With an extra column with the masking information in it. But it seems like not the right factoring of things because you can do all of that stuff. Anything you can do here you can do with transform columns, you know, construct columns and so on. This is all convenience. So the issue here is just walk Me through what transform missing what its syntax is. Should I bring it.

Up? Yeah, if you scroll up or if you open the documentation pages. Yeah, yeah. In.

In.

Yeah. Okay, so is tabular data comma spec, which is, you know, mean median mode, which will apply to all the columns, or a list of.

Columns. Could we please get rid of these nonsense.

Tables? Just delete it if you don't like. I mean, it was a placeholder. I can also do it myself. Okay. Okay. It is a placeholder. There's going to be uniform illustrations in every. In every Tabra page. I can remove.

It. Come to the monthly documentation meeting, which. There's one this.

Afternoon.

What if it is in the afternoon? It's typically very late for me, so I don't think I was ever.

Invited. I think somebody from this team should come to it because I want to raise this question about these, these tables and make sure we have a standard tooling for them. Unless, Roger. Unless you have already done.

That. Yeah, I'm developing it with Tim. You know.

Those. Have you been coordinating with.

Brian? Brian?

No. Well, you need to.

What. What would he.

Do? Because they're responsible for the documentation.

System. He can't make illustrations. I.

Mean. That's right, but we need to make sure that it's uniform, not just in functions that you're dealing.

With. Okay, but. Okay, so I have not found that we can be uniform across the whole.

System. I understand, but I think that you've got to give some visibility to the people who are responsible for the documentation system, for the structural change that you are.

Making. Okay. Can we do it after the.

Project? Maybe you could come to this meeting this.

Afternoon. This.

Afternoon? Yes. 4:30.

Central. I can't.

Sorry. Then we'll have to record it and you'll have to. We'll have to make sure that you listen to.

It.

Okay. Okay. So image. Image aside, let's ignore the image for a second. So you were asking about the.

Syntax. I need to get this added to the agenda appropriately. I.

Wish. When they're prototypes, they're tentative or something that's just, you know, I usually mark them just so you don't fly off the, you know.

Ringer.

Right. And say that this is not. Yeah, it should be tentative and it's like usually xxx before I said this is prototype illustration, not the real.

Thing. Okay, Julio, can you take care of.

That? Yeah, yeah. I'm just going to remove it. I mean, this is something that I put here just to fill in. It's not. Was not designed by anybody. It's Just.

To. Don't need to remove.

It.

Okay. Because we still need a.

Prototype.

Okay. So don't.

Don'T. Okay. I'll just mark it as.

Tentative. Thank.

You. Right. I'm not going to do it because I don't want to have the whole flow of.

The. Right, right, fine. No, no, no problem. I'm going to do it after this.

Meeting. Okay. All.

Right. Okay, so syntax here is quite easy because you already know what is missing. So it's just basically tabular comma, what? Or tabular comma list of columns and each column what to.

Do. Okay, so what we need, if.

We had the analog of that, if there was an argument that says what's anomalous? And then tell what to do. You know, it's one more argument for anomalous that needs to say where it's.

Not. Right. Because here you're separately transforming multiple columns. Column A, column B, column C all have different.

Specs. Yeah. And you could do.

That. Yeah, I don't see any choice given this syntax to having, you know, transform anomalies basically look like, you know, forget this because it's not going to work. You've got to be able to say A arrow, Z score, arrow trim. I.

Think. Well.

That'S. I think Roger was saying A goes, A goes to Z score and then comma, A arrow trim. So you separately. So basically the second argument is calling like find anomalies and generating the flagging mask. And then the third argument is actually basically the same as transform missing. Correct.

Roger, I was really thinking that it would be just like a missing mask, the optional second argument and then the third one that says what you.

Do. But can I just mention, if you're transforming anomalies in a tabular, either they all have to be trim or none of them have to be trim. Right, because we'll trim.

Some. No, no, I use trimming the wrong way here. I feel like trimming essence of reducing the value. This is the one that is clipping sort.

Of. Oh, sorry, sorry. Yeah, I got.

Confused. Yeah, let's. Let's call it like winterized clip.

Or. Yeah, okay, but so I don't understand. I thought what this means is look at column A, you know, based on Z score, things that are anomalous according to Z score, clip.

Them. Yeah.

Correct. Okay, so any other syntax that I can see either has to repeat the A or make a big complicated mess of something else. I'm not seeing any.

Alternative. Furthermore, I think that clip is somewhat context dependent to the Z.

Score. It.

Is. Yes. Well, okay. So then the question is. Is really what we're doing just saying Z score clip just have a. You know, if there are a limited number of these things, you know, we just do.

That. Oh, so names that combine detection.

And. Because I don't know whether that's what's relevant but it seems to me that it could get very specific. Can you give me an inventory? Where do we have an inventory of what we're actually looking for.

There? Well, if you open open find anomaly anomalies. New. But there's just. You know I have like three methods basically univariate to detect anomaly.

Anomalies.

New.

Exactly. Capital.

Nu. It's the same. It's capital sensitive. No, it doesn't matter. It's case.

Insensitive. Oh.

Okay. I would add if you are on a case insensitive type system, the.

Precedent for the combined names is for numeric array where we have things like check and. We have clip and check and clip and.

Coerce.

Right? Yeah, but this is. I mean the point is that. I don't know, it feels like combinatorical explosion. Maybe it's not. Maybe it's.

Not. But just tell me where is the stuff? Where are.

The. Look at the possible values for method include. Yeah, there's a pre unified method plus the stuff that was there before plus the function.

Syntax. Okay, but what do you do with.

Those? You flag examples that are missed that are non misident.

Anomalies. Sorry, no, I. Okay, but what do you want the actions to be? Do you want them by default to all clip or what do you want them to.

Do? No, I think that probably the sanest default is to send them to.

Missing.

Okay. I mean the point for anomalies is that it's a lot more open than missing. There's a. You know, missing. There's a small collection of methods that is. That is like greatly the most used and to find them is quite easy because they are already there. You're just. Is a part thing here is a lot more dependent on the semantic of your. Of your.

Data. I don't understand. I mean like with zscore I don't understand. You know how many sigma are you going to decide as an anomaly.

That mean by default 3. Yes, obviously the node version I haven't pushed actually in my last one. It's similar to.

The. Do we have. I mean do we have something where we go Z score arrow five arrow.

Clip. No, no, I use the same. The syntax I have right now is this like least Z score comma.

3.

Why. Which can be at.

Me. Why is this failing? What is it.

Doing? Failing to do what?

Sorry? The local kernel has quit during the course of an evaluation. Why did that happen? Look at. Just sort of spontaneously quit. Just saying. Who knows what's going.

On? I don't know, but this is prototype, and the things I experimented with today were pretty unstable, so.

Maybe. Okay. All right. Okay, okay. So you're saying there may need to be more detail. How are you proposing to give.

The. Just put a list, like, list Z score comma 3. 3 is going to be the default, and then you can say Z score comma 5 comma 2 to be more lenient. I know you don't like the list thing, but it's a very convenient way to group arguments.

Together. I understand. It's kind of a. It's kind of a cesspool type approach, in my view. I mean, it.

Doesn'T. What. You know, what do you mean by.

That? Well, I mean, it's just. We're throwing a bunch of things in there and it's just like, not. We're not really saying how it all relates. We're just sort of saying, hey, there's a bunch of things here. You know, sort them out. It's.

A.

It's. Okay. This is not the most. The, the issue here is, I mean, I. I would imagine. That in principle we could have a fun function here and, you know, an action function.

Here, A detection function and an action.

Function. Yeah, yeah.

Yeah. In the open case.

Yes. Okay. And the default action would be make it.

Missing. That's very reasonable for usage in transform columns and stuff like.

That. So we say missing, you know.

Quotes, missing anomaly. I mean, in. In fine anomalies, we return missing.

Anomalous. Okay, fine, then that's what we should do.

Here. Yeah, I, I don't see any reason to change this.

Honestly.

Okay. Do we support like different flavors of missing in tabular? I thought we were kind of colder on.

The. I don't think we care. We're not. We're not doing different things with them, but I think we are, you know, we are able to process them. Are we not.

Now? I mean, I guess the question comes up, if you're multiplying a missing not available with a missing anomalous, what happens? What.

Wins? Well, missing not available is not never going to be anomalous because it's missing already.

We. We decided on that. And, and basically, so. So we support. We support the different flavors of missing, but subsequent transformations. Just convert them to the generic.

Missing.

Okay. Or missing not available. I think Was what we decided was.

Generic. Okay, okay, so what do we still have to figure out.

Here? Would there's a. The multivariate problem here. But I guess I think we.

Should. The other one. I think. Yeah, let's not try to get. Look, most of the time it's like I want to do. Okay, here's my point of view. Most of the time, if you're going to use, you know, transform anomalies, it's for a quick, you know, just get rid of the anomalies. If it's like, oh, if this thing is an elephant, then it might not have a, you know, the type of bird might not be the same as if it's a bird in the tree. Blah, blah, blah, blah, blah. By the time it's that complicated piece of logic, your, you know, you shouldn't be using transform anomalies. Transform anomalies is a quick kind of just get rid of the anomalies. If you want to have a very complicated logic about the anomalies, then just do, you know, generic functions and logic and so on on.

Things. I completely agree. And it's the same logic we had with transform missing you and you, you can fall back on transform.

Columns. Can I have a, like a counterexample of this? Like we today, while we were trying to design this, like writing the multidimensional version of Z score with like the inverse of the covariance matrix and then, and then the norm of the vector and so on. It's like it's going to look pretty ugly in transform.

Columns.

Good. It's not something that you figure.

Out. That's why you're writing a version of it. That's just one word.

Yeah. You have to figure out the multivariate.

Case. Sure.

Yeah. But if you use a different.

Name.

Yeah. Just call it multivariate Z score.

Then. What I'm saying is that by the time you want a variant of that, you're going to have to write out all the code. You know, if it's like the dead standard use, you know, subtract the mean divide by the standard deviation or whatever it is. People know how to do that. If it's like subtract, you know, the ktosis from the this from the that and do this, you're going to have to write.

Code. Oh sure, absolutely. But my point was just that we, we have to figure out one complicated thing which is the multivariate.

Syntax.

Okay. We're like selecting.

Multiple. Multiple.

Columns. No, no, I don't think there's.

A syntax is equal once you just give list of A comma B goes to Z.

Score. I would think.

So. Just like we do in transform.

Columns not to Z score because if you say goes to Z score, that means something like perform Z score on these.

Columns. Right. So it isn't Z.

Score. Yeah, yeah. It's going to be multivariate Z.

Score. Z score.

Whatever. Right? Yeah, yeah.

Yeah. What is the right name for multivariate Z.

Score? Goja, do you have an.

Idea? What is it called? What is Z score called in for a.

Multivariate? I'm not even sure it.

Exist. I mean, you just standardize the multivariate data, I suppose. I don't think you need to call it multivariate, just use, you know, we don't want to do that. We just said we can't use Z score on its own because Z score on its own does it per column. Well then multivariate Z score as a string can work or what the word not common, but joint C score, that's not terrible. I mean, that might have a certain. That might make it clear that it's, You know, I think if you saw something like AB. Goes to joint Z score, that would be.

Quite. Also highlights that we are actually considering them together, not.

Separately. Right. Which wouldn't be so clear with multivariate, I think.

Yeah. Doesn't joint. I mean, shouldn't we maybe use joint for the case where we consider it strictly a multiplication of the two univariate and multivariate for the case where you can actually use an ellipsoid? I'm.

Like, I don't know. Can Gosha comment on that? Well, but the point is that if you use the joint disqualor, you want to look at the covariance matrix between the two. Right. And you're going to use that. Otherwise you're just going to assume they are independent and you're going to do the Z score on each column separately. The independent one is surely just Z score. Exactly. So that's why I'm saying let's adjoint Z score. Just as we use notion of joint probability or joint PDF for joint distribution for multivariate, you know, case of variables. Okay, so, so that way, joint issue with ellipsoids and things, which I don't think is relevant, but I, I think that joint Z score means take the ellipsoid version. I mean, take the. Yeah, makes.

Sense. Okay, fine. Fine with that.

Yep. Okay. All.

Right. Okay, move on. One, one final thing that came out as a kind of spin off from this, which might or not be relevant. But it's very quick. I mean, the topic is very broad, but the point is very quick, is that we don't in general have a clear syntax to distinguish between list of tuples and tuple of lists. So one case, and I don't know, Roger, if this was already solved, then forgive me, please. We have this index tabular, and you extract the two columns, you know, and with this syntax here, it basically means a list of tuples. Yeah. You want to consider a list of pairs from A and B. Yeah. Now what if you want to plot multiple sets, you know, like what I had before, tabular, all a tabular, or B. Is it something that we. Because in general, this is also came out with multivariate versus single variant. You might want to. To consider columns independently or you might want to consider them grouped in tuples. Do we. Do we care about this.

Distinction? Yeah. And if you go look at Lisplot, you'll see that how it's documented.

There.

Right. Oh, is there a possibility to do that with tabular? Okay, maybe. Okay, sorry. Maybe I had an implementation. Sorry. Then, yeah.

Then. Oh, no, wait, wait. There is another thing. Sorry. Like, we transform anomalies. Do we allow for, like, do this thing, the anomaly mask or whatever you're doing, the send to missing clip, et cetera, but instead of like overwriting the existing column, write it to a new column named.

This. No, no, it's impossible again. I mean, if we go down every path, the syntax we'll have here is unbelievably complicated. Nobody will ever understand.

It.

Exactly. I think most of the time, if you want to transform anomalies, you mean zap those minus 99s and turn them into zeros or something, But. Okay, okay. And a lot of the times you'll just be like, let's say if every missing data is -99, you'll probably just want to say, transform columns -99, arrow missing. Makes sense. Yeah, let's not chase weirdo syntaxes. They're things that people just want to do as a kind of like, oh, I want to Windsorize my. You know, whatever. Just boom, they do.

It.

Exactly. Yeah. All right, ready to move.

On?

Yes. Okay. Out of core. No ferrite is being harmed in this.

Process. Too many.

Physicists. Well. Or too many old people. Yeah. Okay, what are we seeing.

Here? So, yeah, so, I mean, now, Rob has provided me with a number of things to change the boxes, to implement everything. So basically, now, if you open up the flipper there, this is what an out of Core tabular looks like. No, that's relational database object below the one with head tabular. So it has the backend, the column count the name of the columns. If one is a key column, then yeah, that's the unwashed stuff you don't want to see from, from the SQL.

Zoo. Okay, so you can.

Tell?

Sure. I think this should be grayed down. Why is that one bold.

Face? Because it's a key.

Column. Yeah, I thought you were going to say that. I kind of think a dingbat would be better than.

Bolding. What.

Dingbat? I don't know, a key. How about a key as a.

Ding? I, I, I'm afraid it's going to be small at that font.

But. I know, I know. Why are we explicitly saying the word columns here given that we just said column? I mean, I'm just, I'm just talking about formatting.

Here. I mean those things have to have a header of some sort, if I understand.

Correctly. I don't know if you know, that's just because of the way that those.

Box. I also don't particularly like that it's center lined rather than toplined.

But. Yeah, well, so complain. I mean it is not here. It.

Is here, but.

Muted. That's unusual. Itai with us, he zoned out well, anyway became an.

Anomaly. Okay, so you want the string to be grayed out and.

Then. Okay, I think the word string should be grayed out and I'm not even sure that we shouldn't put it in parens after these names gray out in parent because I just don't think.

It'S as so not, not.

Agreed. Just I don't think so. I mean, I don't know what other people think, but I, I don't think. Well, I don't know. I guess I can see arguments where you want to read down the list of.

Types. Yeah, I think having them aligned is.

Nice. Okay. How complicated can those type specifications.

Get? No more than that. Like we don't have integer 64. We hide that from you. Really? Because otherwise it gets too much of a zoo with different.

Vendors. So I'm sure we don't want to do those things as like icons of the same kind that we have in the actual. By the way, these icons have not been fixed, which kind of annoys me. Right. We went through that like a month ago.

Right. We had a meeting with Jeremy and Tim and Jason and we got another revision that we, I expected it to be show. I think it has been.

Showcased. Well, I haven't, I don't remember Seeing it. But if we've got reasonable icons here, why not use those icons here? Maybe even on the left, right. And then we can also have.

Like, if there is room for, for text, I think it's better than having to parse an.

Icon. Like.

Maybe. I.

Understand. I don't think these icons are that important. I mean, most of the time you don't even care what the type.

Is. No, I strongly disagree here because in the case of this, it's more opaque and it's more strongly typed, so you can't add the number to a.

String. My suggestion would be loudly, both graphically and otherwise, that here's a way we could do it down the left hand column we have the icons. Key columns have their icons bolded. See what I'm.

Saying? Yeah, but we have the room for text. To me it feels.

Like. Doesn't mean we should use.

It.

Especially since we only have three or four.

Cases. Yeah, but the thing is that unlike the rest of Mathematica, Sorry, Wolfram language, when you're running this on a SQL database and you have a type mismatch, it's going to fail loudly. And so you. This is like an important piece of information. Has your.

Typing. It's already failed long ago by the time you get to this point. It didn't.

Fail. No, because like, for example, the Office code looks like numbers and you're like, oh, okay, I'm going to add one to them. And then, oh, no, they're.

Strings. Like, I don't know. I don't, I don't have. What do other people think? Think? What does Christopher think, for.

Example? I don't really have a very strong.

Opinion. Anybody else has. Have a strong.

Opinion. I like them being written out and explicit. Knowing what the types are is often important to.

Me. Well, given your particular job, I'm not.

Surprised. Yeah, fair enough. I like seeing the types too. But that's, you know, that's because I work with types all the.

Time. So isn't this like one of those basic rules of ux, like if you have the room, like wasn't Apple slammed a number of times for removing writings under icons or Microsoft rather than Apple, that it's all icons, you have no text where there would be space and then it's incomprehensible after when there are too.

Many. I don't have the one question is.

When? When would you be trained on what the icons mean? If it's never shown icon with text next to.

It by seeing. Yeah, it is a good point. All right, fine. Let's have text. Write down, but let's have text. And now in terms of the key column, I would put like a dot before it, center dot before it. Okay.

Okay. But I. I think that for the entity store we used bold.

But. Okay. Really?

Yeah. Well, I can change that too, if you want. For the. It's canonical name there.

But that's a little.

Different. Maybe it's underlined. I don't remember. Let me.

Check. That's a little different though. Canonical name is a little different from, you know, key column has very different functionality. I mean, canonical name is. This is the thing you should pay attention to. Whereas key column is much more of a, you know, in the weeds operational.

Thing. No, no, no. Canonical name should be opaque. It's not something you should pay attention to because it's like it's.

Something. Okay, we'll find out how it works though. Yeah, okay. What else do we have to do.

Here? Oh, okay. There are a couple of open questions that we left off last time we talked about.

This.

Yes. Tabular twin memory is obscure. Should be a single function. I think we need to name.

It.

Yeah. Okay. I think that function should be the same as the function that brings like a video in.

Core. We can't. We don't bring video in.

Core. Not yet. It's small enough. We could do that, right? Audio we can bring it. Can we bring audio in.

Core?

Yeah. Audio we.

Have. Yes, we do that, but only for audio, not. Not.

Video. What's it.

Called? It's called.

Normal. What's.

That? Normal. Normal of out of core audio gives you in core.

Audio. That's a bit.

Overloaded. Yeah. And also there's the problem that normal is usually item. Potent.

Right? Yeah, I don't think it's really the right name for audio. I think that was a sloppy piece of design which I might have been responsible. Responsible for. I think we need a function that brings things in core. Let's think about what else we might bring in core. Okay, so. What we have right now is local cache, which does not bring things in core. I mean, I could imagine somewhat fictionally perhaps. I mean, don't we have. I mean, we haven't used these. Do we have street in course.

Streams? What do you mean by.

Streams? You mean.

List. No, no.

No.

Stream. Yeah. Yes. Is that an in core stream or is that. It must be an in course stream. It's not a.

Thing. I don't know what you mean by in core.

Stream. I.

Mean. I mean string to stream, I think doesn't reference any file.

Right. Okay, then it's an encore. Because what I mean is, look, if you have a gigabyte file and you're doing seeks in the file, okay, that's an out of core stream. It has a stream pointer, and the stream pointer refers to something that is out of core. So that's another case where if we have a mechanism, not that we use this very much, but we're bringing the stream into memory, which is relevant if we're seeking around in it all over the place. And you don't want to be, you know, doing that in a file.

System. Well, that's like read or binary.

Read. Binary read operates on the file system. Yeah, yeah, Binary read operating on an incor stream is what I'm talking about. And that is a thing, right, Sean? That. That's a thing. Yes. Okay. Do we use it.

Often? But I mean.

Internally. Okay, okay. So I mean, that would be an example. This function that brings a thing in core. One could imagine it taking a file and making it. Or, or a stream for that matter. Where we've said open read and it returns a stream, we can imagine in corifying.

That. Well, actually, import already does that. Import does work on input.

Stream. No, I, I understand that, but I'm saying, I'm saying, I'm saying identically bringing. Yeah, yeah, maybe it should be import. No. For a second in some.

Format. Exactly. That's what I was just going to.

Say. What I'm suggesting is whatever this, in corify, whatever we call it, that what it does is. I mean, this is so, so.

When I'm working on a file or something, it's like read byte array or read string. Brings the whole file into a string or byte array. That's how I bring something into.

Memory. Yep. But what I'm saying is if you've got, if you have a thing where you say open.

Read. Well, there's a developer version of this thing that's like string from string. There's some internal stuff that we do to do that, but it's not designer exposed at.

All. Okay, okay. So what I'm saying is, I'm just trying to get a sense as we try to design this in core, ify function, whatever it's going to be called, I just want to get a sense of where else it might be used. So that's an.

Example. You.

You'Ve. You've done an open read and it is. What do we. What is a temporary file? Right. All right. Temporary directory. How do I get a temporary file $temporary directory. But what, how do I just get open a temporary.

File. Open.

Temporary. That's what I thought. And I didn't seem to know about that that. Oh it did. I'm confused. Wait a minute. I'm going crazy. Look at that. I'm not going crazy. Where's the autocomplete? Am I going crazy? Hello.

Everybody. It's deprecated in the docs. It says you should use open. Right Open right open close. Does what open temporary used to.

Do as of version.

6. Okay, that's what I'm talking about there. Okay. And I'm saying to incorify.

That that's an output.

Stream.

Yeah. Well, okay.

Fine. Which is why there's like a string from stream thing. I. I don't think I like.

When you keep talking about ink correfine.

Strings. I. I think I'm not sure how well formed this is from.

Like. Yeah, I mean I'm just gonna get a scope of what you could in correfine. This is one, one other thing. What else can you imagine A database? Is that a thing you could.

Imagine? You could, you know, you could take an out like a database backed entity store and turn it into any memory entities or probably blow up your.

System. But you can try. No, maybe it depends on what you're.

Doing.

Right? Yeah, if you. Okay, so let's make a list of some of these things. Okay, so we've got entity stores, we've got, we've got audio, we've got video. What about search index or a vector database? Does that make.

Sense? Guys, Data connection object, something that's pointing to like an SD file or something. Something, some remote resource. Data connection.

Object. Wait, is that like if it's on S3, you're going to make a local copy or you're just going to bring all the whole S3 backend in.

Memory. That can be a bit of an implementation detail, but if you have some S3 file that you're pointing to a specific object, you want to bring it into.

Memory. Yeah, I mean the whole point here is if you've got a terabyte of memory on your computer and you know you're going to be operating on this thing, you might as well bring it into memory. So what it probably should be called, since we call memory, we have the notion of memory and use memory available, all that kind of thing. I think the, the key word that we want to use is memory. Right. So to memory of expert. Now what's it going to return? It's normally going to return something which looks the same as it did before. Is that true? I mean, yeah. You understand what I'm saying? I mean this two memory of an outof core tabular will just return a thing that says tabular. Right. Which now happens to be an incore tab. Tabular. Am I right? Can somebody say please? Okay, is. Is anybody.

There? I think so, but I'm thinking.

Hard. Okay, all right.

Fine. I think, I mean that's exactly what two tabular two in memory did down there. Right?

So. Yeah, I think.

So.

Yeah. I thought we could call it wolf Ram. Very.

Funny. So. So for the, for the two I.

Thought. I.

Don'T. I remember we talked for a long time about how to bring tabulars into memory and I thought that we made a new function because there was a discussion about it could be to tabular and it could also be from tabular that brings it into. You.

Know. I.

Know. So those, those are. Yeah. To tabular is basically a converter from other expressions where you can control how you make that conversion. From tabular is a converter from tabular to other expressions where you can control how you make that.

Conversion. Why can't. If you're going tabular, if it's an out of core tabular, why isn't it just two tabular with one argument that was suggested that like we had a long discussion about this at some point and that was suggested, but one of the questions was it could just as well be from tabular because there's a tabular on both ends. Right. Because it's.

Going. The other thing is that there might be a day where, you know, we want to take an incore arrow compatible tabular into an out of core parquet tabular or SQLite. And so it would maybe one day we'll have the opposite thing. So to call it just two tabular with zero arguments will just.

Just. Yeah, I don't. No, that's a great point. We really need to have a name for something that's in core and out of core. Both of them need to be explicitly named, I.

Think. Right. Well, you could do that with two tabular and then, you know, look, if it's two tabular and it's going into something in memory, what are the typical second arguments to two tabular? They're not where it's coming from. Right. They're where it's something about where it's going to. Is that right? Or am I.

Confused? No, two tabular is where it's coming from. Like two tabular comma columns means you're specifying an association of columns or Say time series means you're specifying a time.

Series.

Yeah. This is also why it doesn't really fit here because like in general, both from tabular and to tabular, specify the thing which is not tabular. The tabular is.

Like. All right, so I, I am proposing to.

Memory.

Yeah. And this thing that you're talking about, about moving something from memory to an external environment, sounds to me like an.

Export. Yeah, it's probably is like given the number of different parameters you, you can have.

Yeah. All right, so to memory then is essentially an optimization. Now okay, let me understand this. What is the entity? Entity prefetch. Right. Is a very analogous.

Thing, but I don't think it goes to.

Memory. No, it doesn't go to memory. Yeah, doesn't go to memory. Although one could use two memory on an entity.

Type. I think you can use it on an entity.

Store. Yeah. Okay, but I mean you could use it on a, on a built in entity type if you wanted to cache a bunch of things in.

Memory. You'd have to ask Tony if that's.

Possible. Well, it may not be possible now, but it's certainly something we can imagine. I'm just trying to understand what to memory could possibly do. And it seems to me like it's a fairly generally useful function, but that normally what to memory would do is do a bunch of, you know, routing around, but then it will return like local cache, for example, or like once all of those functions do something, but then they return the thing. They return a thing that symbolically looks like thing you started from. Am I making.

Sense? Yeah, I think that we're dealing with is like a reference to something else that's not already in memory, but there being a.

Reference. Look, here's the confusing thing, okay. You have a not in memory tabular. It will look fairly similar to just a generic tabular. It might have different iconography and things, but it's, it's basically looks like a tabular. Okay. Then you say to memory and what do you get back? You get another thing that looks like a tabular and that has, if you were to look at it, it would have the same data as the you started from. It is an equivalent expression to the thing you started from. I mean more efficient because it's in.

Memory. I mean maybe this is temporary, but right now those outer core ones come with a summary box and then do they do not look like.

Tabular? I, I know, but they look somewhat similar to tabular. I mean, I, I don't know how similar they look, but they it's fine. I mean, you know, if we say two memory and it comes back and it's a tabular and the thing that it was before was an out of core tabular, everybody's happy. I'm just going to clarify what 22 memory returns. It returns an object which is functionally equivalent to what it started from but just more efficient because it's in memory. Do we.

Agree? I don't agree it's necessarily more efficient.

But. Okay, okay, okay. It's in memory and it's. But it's the same thing. Like you can do an operation, you can count the number of rows, all that kind of thing. The operations that would worked on it before will work on it again. Yep. And that should be true of the other examples of 22 memory which I mean we should probably talk about this in LCC with a different set of people for some of these other things. But I think entity store will be a very obvious one. I mean this doesn't all have to be supported on day one. Make sense.

Everybody?

Yep. Any objections? And I like the fact that you can say to memory, you know, you basically say memory available, blah blah blah. The only thing I can imagine that we may or may not be able to do is a two memory pre flight. Do you understand what I'm.

Saying? How big will it be once I put it in memory? Yeah, yep. Yeah. We could, we could have a limit above which it would somehow give a.

Failure. Well, I mean if it's bigger than memory available, it should give a failure. Yeah.

Yeah. I just don't know. I don't know if I can do.

That. I mean there could be an option that we. It controls a limit the fraction of memory available that you can consume. Like if you.

Go. But I'm more worried about like.

I mean if you fly, whether you can tell before you do.

It. No, I, yeah, like performing a query that says. Okay, let's give me, give me a rough estimate of how big this database is. First of all like you're forcing it to do two round trips to a database. Secondly, if it's something that has like 15 columns and you have to check all the types and then check the length of the.

Thing. Can I make a comment here? It's kind of obvious. If you can, if it's going to live in memory, you better be able to do those things. If you can't do those things in reasonable time, it's going to blow up your.

Memory. Not true, not true. Because let me give you an example. You have like A super complex query that ends up returning one row with three columns. It just takes a long time to compute it and figure out that it's going to three.

Columns. Okay, let's, let's just comment on what we, what we're looking at here. Okay. Somebody has a database. They wonder, is it plausible for me to bring this into memory? Is it 100 megabytes or is it 100.

Petabytes? Okay, yeah, but Stephen, again you're thinking about like what I'm saying. Imagine you have a situation where you have like a database that has, that is tens of terabytes, and then you're filtering for a very, very rare case that will return three rows. But in order to figure out if there are three rows, you have to run through every single case and it takes five minutes to.

Process. I understand you have to perform a select. So, okay, but I'd like to know, if I gulped the whole database, how big would it be? Now, maybe there's a way to get that from the relational database.

Function. No, but that's something you can reasonably tell. But most of the time what you're going to do.

Is. I don't care. I don't care. I'm asking. Here's the point. Here's the point. Okay. If you say how big would it be? If I gulp the whole thing and the answer is 10 megabytes, then whatever you do is going to be.

Safe.

Yes. If you say if I gulp the whole thing, how big would it be? And it says it will be an exabyte, then you have to worry about it. Right. So being able to quickly check roughly how big will it be is useful. Now, if that's something that we could provide in the summary box for the relational database, that would be okay. I mean, you know, there's a certain byte counter.

Yeah. I think that the answer to this question is in General, if it's SQLite, it sits on your hard drive, you can tell and it's probably fine. If it's sitting on some remote server and it's postgres, then most likely it.

Isn'T. But not.

Necessarily. But because it could be the network latency that makes it slow, you don't.

Know. The point is you might not know until you actually execute it. So you might as well execute it with a threshold where it fails to pass that threshold and then see how big it.

Was. Well, see how big you got to before it failed. Sure. Okay. Because here's what I think is actually a pretty common case. Somebody has had a database lying around for 30 years it's a database of turtles caught in the wild until ducks are a better animal to use. But anyway, the, the. And you know, at the time when it was collected, that was a big database, you know, but now it's, it's turtle, you know, it's tiny. The. And you know, but you just don't know. I mean, people, I mean, I, you know, you just don't have any idea. And being able to quickly check that will be very useful rather than doing a thing that most people were extremely reluctant to do, which is to say, just try and gulp this. You may turn out it's, you know, you're eating something that's 75 times your.

Size. I think that the proper solution for this, which also prevents you from like doing like the, the operation multiple times, is to have support in external object or some such to cursors. So, you know, we, we can.

Set, we don't have to have that thing. We're just trying to answer the question. This database that somebody handed me from whenever, you know, or somebody is pointing me to, how big is it roughly? If it's, if it's a file, then obviously I could in principle go look at the file size. But I'm talking about users who do not understand. Is it a file? Is it a remote thing on a server? What the heck is it? Is it from 50 years ago and it's tiny? And you know, whatever it's like, couldn't we just have a function that just tells you the answer to.

This? I'm not even sure that like all sets of permission lets you see it. But I can check. I can see what I can.

Do. Okay, but let's just think how we would design it. If we could have this, okay, Would we simply put it in the relational database box or we have a separate function like bytecount that tries to figure it out. And by the way, one of these years or decades, we need to make byte count more sophisticated and deal with external objects and all that kind of thing more effectively. I mean, it's not really been important or really come up. But this is a case where as soon as we have outer core things. If you say byte count count of some video, what's it going to do? Am I making sense? Do you understand what I'm saying? Somebody.

Please. I think it gives you the bite cut of the expression which is very.

Small. That's what I think too. But the fact is it's not necessarily what you want, Right? Am I making.

Sense? Yeah, but that simpler case databases you have a file most of the time.

Local. I know, I know you have a.

File. I'm saying take a look at.

The documentation for byte count. Though it does say rather specifically that it's what's in memory in the system and it is the memory of the.

Expression. Well, I understand, I understand byte count.

Was. But I see what you're saying that that's not what you actually wanted. What you did want was the size of the.

Video. Right. I mean it may not be the, you know, it would either be byte count with another argument or an option or, you know, I can see that you want these remote byte counts in certain cases and that's the same thing we're asking for in all of these cases. Now notice that if I say file, blah, blah.

Blah. I mean there's something else to be aware of, like audio and the same for video. They are stored in compressed formats that are really optimized for playing but not processing. When we're in memory, we don't store them in those compressed formats. They are in a format that's optimized for.

Processing.

Right. So kind of similar on the database size. Like you don't know the size of the query is going to be proportional to the size of the database. Like they probably are, but not.

Necessarily.

Right. Yeah. One way to think about it is really like whenever you have a query, it's more of a computation than an import. So you could have a query that returns a really large thing from a database that is zero sized by doing like recursive stuff. It's, it's a. It's true. Incomplete really.

So. Or this is a joke, but just one cross.

Join.

Yeah. All right, we should probably wrap up here. I mean, I don't think this is a critical.

Issue. There are a couple of questions if, I mean if we want to use the full extent of.

The. I'd love to not do that.

But.

Okay. I mean we don't need to use time that we don't need to use. But if there's something that is holding anything up, then we could talk about.

It. I mean, there are a couple more questions like in terms of like if you remember.

Chain. I'm sorry, I'm sorry, I'm sorry. I'm going to bring something else in. We probably need a parameter for now. Max. I don't know, I don't know what the name of the parameter is, but. Something like that and a corresponding option. Did you understand what I'm.

Saying? Yeah, that makes sense. And also maybe a time constraint.

Type thing yeah, it's a good idea too. Now actually. Also, we need to make this work with memory constrained. Right? I mean that will be another way to do it, but I would consider that less reliable. Do we have any other max memory conditions like that? Do you understand what I'm asking.

Anybody? Yeah, we've tried, but they're, they're inconsistent in some sense because of the way different.

Platforms. What have we done? What have we done this.

With? I've tried to do something for sparse, Ray, but that's not even.

Documented. So when we go to dense, you.

Know.

Right. So when you go to.

Dense, so you don't convert a million by million.

Matrix. Right. But the default for that is infinity. So it's not really, it's not really.

Used. But what is the.

Option? It's, it's a dollar something and I don't honestly remember what it was. Let me.

See. Is it not documented? Yeah, it's not documented. Is that.

Correct? It's not documented. That's.

Right.

Well, Okay, it's, it's, it's sparse array. Back tick $sparse array normal.

Limit. Actually, when referring to memory, we usually use size limit summary box Data.

Size limit, output size limit. I see. So if we called it two memory size limit and then $2 memory size limit. Or just size limit and then we could use that same option, dollar size limit. We could use that same option in sparse array or we could call it memory limit, which might be.

Better. Well, that it might actually be a good option to.

Normal. Exactly. Yeah, yeah. Why not? Memory limit $Memory limit for places where you blow memory up and you know, it can default to infinity if we, if we're worried about it. I like this. Yes. So then two memory has a memory limit option. Seems to make sense to me, Roger. Anybody.

Else? So, so wait, I'm, I'm a little confused as to. What are you thinking of just memory and dollar memory or both size limit memory.

Limit? No, no, no, no. I'm, I'm, I'm just. Those are alternative names I'm.

About. Oh, oh, okay, fine. Sorry, I got a little bit twisted up.

There. But yeah, that's what I'm suggesting. I. Accepting that the function is two memory and that to memory has a memory limit, but memory limit also occurs in. The only thing that I wonder about is whether we should call this something like expression memory limit or byte count limit. Because I could imagine wanting to stop the system if memory gets above. If memory available goes below something, for instance. Am I making sense? Other people want to comment? Christopher? Roger. Anybody? Jose? Anybody want to comment on this.

Name? I think if we introduce it as experimental, I think it's pretty harmless at this.

Point. Do you see an objection? The question is whether we differentiate. This is the memory limit for an expression. Versus memory limit for the. The. The, you know, you know, memory used. We could probably have something based on that if we're really going to go. Because I think it's a much more undefined.

Thing. Yeah, yeah, I think we. I mean, I think memory limit seems fine because we've got, you know, memory in use, max memory used. And those are all referring to byte count.

Basically. No, I'm not worried about it. What I'm concerned about is memory limit refers to a single expression that you're gulping into memory. Yeah, right. Whereas memory used refers to the system's total memory usage. So the question that I'm. But we have memory constrained, which refers to the memory of that computation, that point that I think wins the case. Yeah. All right. I'm good with memory limit.

Yeah. And that could be an option to normal, too. You're.

You're. Exactly. That's what we just.

Said. Yes.

Lovely.

Okay. And what other function. Whatever other functions, you know, puff memory out normal, I think, is the number 1 1. And then for audio, we would start supporting two memory, and we would deprecate normal of audio, probably. Am I making sense? Sean makes.

Sense. Yeah. There is a note in the ref page, so you would have to change.

That. Yeah. Okay. Okay. Anything else, guys? Okay, great. We're making progress. Thanks a lot.

Everybody. Thanks, bye.