Live CEOing Ep 799: Language Design in the Wolfram Language [Tabular]

Okay. Hello, everyone. Welcome to another Wolfram Language design review meeting. I am told that this is the 799th episode of our live CEOing series. So almost. Almost. The next one will be our 800th open design review live stream. All right, let's get on with the 799th one. We are talking about tabular. And where, where are we starting?

I'd say with element types. So we are discussing the variety of types that we can have in columns and how to change types, etc.

And how do users ever see this?

Actually, that's another thing we should discuss. So if you are in repl systems like Python and R and so on, they. That's kind of. When you summarize the data frames, you get the types. It's very long and verbose. If you're in commercial systems like Tableau.

Why are we talking about. Why aren't we talking about what we're doing? Why do we give a. I mean, I'm sorry, let's talk about what we're doing.

Yeah, so imagine that the user.

Well, I want us to see the types of. Or some version of the types, because.

Okay, well, that's a lot of concentrate on and that's what we care about. Okay, so you're saying it's useful to.

See the types, not that this level of granularity. I don't think anybody cares whether it's integer 8 or on site. Integer 8.

Oh, yeah. So what is your point? What are you suggesting? Do you have a proposal or are we talking about every other possible system that we don't care about? I mean, is there a proposal here?

I think that there is a proposal and that's how to set and cast the types.

Okay, yes, let's talk about that.

But you asked how you ever see them. I say, well, I think we should see them.

Okay, but that's what I want to hear about. I don't really need to see. Hear about what every other, you know, what every other bad design decision that somebody has made is.

So I think once user has data, it would be important to properly tag that data with the proper type. And there is this variety.

I mean, obviously if you're using a, you know, a database system from the 1970s, you're going to know about types because that's sort of a central thing there. But that's kind of not what we're going for. So the question is, why do people need to know about these types and what are they going to do with them?

Right.

Well, one, one question is how often I mean, maybe, maybe someone who's been playing around with this can say, like, how often can you automatically determine the types, like completely correctly? Like, how often do you, practically speaking, need to intervene?

So I think that it's a cleaning operation to set the types right. Because. Yeah, why you want the types is for downstream decisions. Like if it's, if it's a number, there are certain things you can do. If it's a category, you can't do arithmetic on categories.

Right.

That's why, that's why if it's a date, there's certain things you can and cannot do.

Yeah, but what can and cannot do, it's a question of you try and do it and it's the wrong type, you get an error message or what?

I mean, I would imagine in a lot of cases it's just inefficient. It's like, for example, if you use the nominal scale or something like that, and it's basically categorical strings or something like that, if you didn't make them, you'd imagine that they're internally represented differently, but when you index into it, either way you just get a string.

That's absolutely correct, Christopher. And that's right. So the categorical stuff, and I don't think we're doing it quite the right way here, but it's basically a different internal representation which is more efficient and it also tells you what the distinct elements are.

So look.

When you do regression modeling, you're not going to regression.

I understand all this. Okay. You're importing a CSV file. It starts off as a great big string. The question is, is this column integer 64 and there's some garbage in it that we want to treat as missing, or is it something that is really should be considered as a string or something and not as, you know, as an integer 64? Is that a fair characterization? I mean, in other words, the data comes in from somewhere. Maybe the data is coming from a database which has very clear types, but maybe it's a kind of an efficiency thing for us and a, and to some extent a functionality thing. And one would hope that the user assistance system, for example, would know about column types so it will be able to correctly auto suggest operations and things like that. Right.

It's also tied to what you call semantics, meaning of things like if the thing is the date string, you can't operate on those date strings that great.

So what I mean, we've got a function, you apply it, it either gives an error message or it does what it's Supposed to do?

Yeah. So what? Right, so, so, Stephen. So data set. So I don't know what's being proposed here, but data set. Every time you create a data set, right. It has its own bizarre type system and it assigns based on its rather poor heuristics, in my opinion. So I presume that if I had a matrix of stuff then, and I wrapped it in tabular, then that's going to look at each of those columns and assign sort of the most respect.

I assume you've already done this, that contains that.

Yes.

Right. I assume this is already done for tabular with some set of heuristics and so on. Is that correct?

Yes, there's, there's an automatic set. But, but, but one, one example, I mean, I think quantity types, you know, as long as it's a column of quantities, that's for. That was just an example to show you. But if you look at the date type section, which is just below that, that shows an example of a, of a tabular that comes in from the wild, okay. And it's in New York Times Covid data set. And the dates are all strings. So how, how do we tell.

Okay, that's a perfectly real issue, right, that we're doing an import of some kind and I don't know what the heck this is. What the heck is that? What is that? What is that?

That's a, that's a temporary example data until we figure out where examples are going to be. Okay.

All right. So for now it's either in the data repository or maybe in example data. Although I would think it's better in the data repository. I would think it would be better to have it simply be a. You know, the resource data will typically just return a tabular, right?

Yeah.

Long term, that's it. We.

Okay, fine. But so in its initial form, this was a bunch of strings. So really this should have been an import of some CSV or something like this. Is that correct?

That's what it was originally, yes.

Okay, fine. And on import, the thing didn't know to convert some column into. From date strings to date objects. Is that correct?

We, we actually, we've changed it, we've added types.

I, I don't care what you've done in this particular data set, but generically you've got a CSV and the way the CSV importer works, it does not yet have a way to say, oh, by the way, this column I want to do this with that column I want to do that with. Is that correct? Well, I'm currently working on this.

Hopefully it will make.

Sean, doesn't CSV automatically detect types and return date types if it recognizes it as a column?

For some of them it does.

So CSV, the old implementation didn't have column wise interpretations, so we weren't detecting column types. It was element by element. Piotr is working on a new CSV importer using Arrow that's way more efficient and does some type detection for columns. And what we want to do is correct it.

Yep, right. That's what we have to concentrate on is how do we make CSV import be capable of getting a template basically of what the, you know, being able to guess the column types we need a flow for. Yep. I mean, you know, we need a flow that either guesses column types and you can correct them or well, you.

Can import the schema and then, you know, if you see that one of the columns.

CSVs don't have schemas.

The inferred schema from arrow.

Okay, but that's not import, right? That's not important.

There are all sorts of properties. One of the things you can ask for is the schema that it will use.

Right, fine. Okay, but so let's, let's understand the flow. The flow is you've got a thing in the wild. I mean, what we really should be talking about here is we've got a certain set of types that we're supporting. We maybe should discuss what that actual list is. But there's a set of types, it will be expanded. We want to be able to say, and some of the types, you know, then we want to be able to say, take this thing that we're importing and cast it into some sequence of columns with these, you know, with types from this set. Is that correct?

Yes.

Okay. All right, so now let's just have a little bit of a survey of what those types are. So okay, there's categories and that's your categorical data. Is that correct? That's not quite. This is it.

Yeah, that's.

Well, that's what we're calling it now, but I don't really think that's the right way to do it.

No, no, no. I mean what we really want is just controlled categories. So here are the five things this thing can be and you have some way to store those categories that's efficient where it's number one, number two, and then you've got.

That's what a nominal scale is.

That's what it is.

Okay, for my simple minded benefit, a nominal scale is an enum.

Yes, okay, but then there's, there's weirdness. I, I, yeah, I, I've never fully understood what the point of. So I know, right? Ordinal scales are nominal scales, but where you have an ordering on the categories. But I never understood what operations you.

Could actually do ordinal have ordering on it because people get very upset, for instance, if you present them as small, large, medium.

Yeah, but, Right, Oh, I see. So it's just so you can sort. It's just so that literally the function order can, can give something reasonable.

Nominal only knows about equal and not equal.

Right, well, but, but now it's a little weird though for us because then suppose you have an ordinal scale, right?

Like, like.

Okay, so, so suppose I tell you off, Christopher.

But I, if I think the point is, reading Rob's mind is that nominal, that a nominal scale only has a canonical order, but an ordinal scale also has a numerical order. Or am I wrong?

In ordinal scale, basically by the order you give the category names in, that determines the order for the whole thing. It determines the ordering function.

So.

Right.

But I guess the question is where that ordering function can ever manifest itself. Because for example, okay, so for example, if I index into this thing, if I get out my tabular column, for example, I say normal, which I guess is probably equivalent to just taking all the indices, then what I'll get out is say a bunch of strings, right? Maybe there was an ordinal, ordinal scale of strings and those strings don't have any order.

You're right, Christopher. So this leads to another thing which I haven't started yet, but I think I'm considering doing a design where these going to have a much wider life in the whole system. So we can have an ordinal element kind of like we can have a date element that has super high overhead with storing calendars and all other stuff. This ordinal element would have a little bit high overhead too, because it needed to know about its other things and how they are ordered compared to them. In which case it would work as a free floating element in the system. And answer your question, where can it be used? Sort would know about it, for instance.

Yeah, unless greater than.

So, yeah, well, we don't have that right today, Christopher, but I think that that's sort of one of those things.

Okay, I mean, but certainly sort. I mean, if we have a, if sort works on tabulars, then it could take advantage of this as long as it's in its tabular column. Yeah, right.

Yeah, well, for example, I mean the.

Visualizations and the modeling funct take advantage of it.

A super good example is sort by. So if I, if I, if I do a sort by one of these things, all it has to do it, it's, you know, it's, it's very quick and then the, the whole, the whole tabular can be sorted based on the ordering of that one column.

Okay, so gosh. So what we're talking about here, I mean I, I do agree that these categorical things, I mean just having sort of authority control I guess it's called in the library science business of saying it must be one of these five things is useful and if it isn't one of these five things, then what do you do? You generate a message, you make it a missing, etc. Does that make sense? Yes, and that's a useful thing on import.

Well, actually, sorry, a quick thing is right. I mean as mentioned earlier, there are two reasons why you'd care about for example, saying these are, you know, nominal scale. All like the term nominal scale is bad, but whatever. Right, right. That these are categorical data. I mean one reason is what you just said, that I want to enforce that the state column only gives one of the 50 states. Or, or but then there's also the reason of just. It's much more efficient. So like there are questions around. Suppose I have a tabular with a nominal scale and it's got, you know, the states there. And then I maybe append. I assume we're going to have some sort of append row mechanism. I'm guessing then suppose I append something and I give it, you know, something that isn't one of the 50 states as my, you know, in that column does it try and be nice because all it cares about is efficiency and it will make it, you know, a new tabular column with 51 category, you know, classes. Or will it fail or will it.

Turn that new thing into a missing. Yeah, those are all things that have to be defined.

Yeah that those, those have to be defined and we need, we need directives to, to moderate, moderate that behavior in some sense.

Okay, so I'm assuming that two tabular has. Yes, has a whole bunch of controls for this. The two tabular is one place where I could take an unwashed array of some kind and into tabular I can give a specification for what should happen to each column.

Yep, we have it, we have as an option to both. But maybe it should just be an option to two tabular and I'd be Option to both.

What to tabular. Well, let's see what the option looks like.

Yeah, so here's an example. I have a year column and it's got dates, which could equally be. Well, strings or integers or dates. Right.

Yes, sure.

So you want to be able to change or direct.

What are we doing with this element wise thing? Have we, have we understood what that design is?

Well, element wise cast. But, but it's, it's, it's. What it's like is if you wrote. So suppose you wanted to change.

You.

Know, from do like a cast to double. Okay. Which, which would be in C. Okay. So in the compiler you'd write a.

If.

If you wanted to do it for a whole array, you'd write a loop and you have a cast inside that loop that did that.

Yeah, I understand.

Here. Yeah.

How far we got with element wise last time we agreed we're going to do element wise. Right.

You have here we agree, we do it and it's. I've changed it to have listable semantics.

Okay, so remind me how I use element wise just in with ordinary lists and so on.

Okay, so you do. An example is like element wise of less.

Oh, right. Yeah. This is a really nice design, I think. Right. Is it going to work? Whoops.

If you've got a fairly new prototype.

Yes.

It should work.

Right. Of say, you know, 1, 2, 3, comma 2 or something or 5. Yeah.

That'S very nice. I mean that's kind of a map thready type thing. But this is probably a cleaner way to do it.

Yes, this is very close to map thread.

Yes, but map thread can't handle a scalar for the second argument, which this can because it's doing listable threading.

Exactly.

Yeah.

Yep.

That's really nice. Really nice. Okay, so let's come back down here to sort of. Do you have. Okay. I mean this is very cish. I have to say that we're having to do this.

Well, there's. So I. I list four different ways of conversion here and we sort of have to decide what. The reason I put something together like that is that. And, and I don't know that Carl actually likes this, but he said for transform function we need something that works. That acts like it works on the elements of the. Of the column.

Yeah, but honestly, I think that if you like for databases, we've gotten away with using all the functions like N or floor, which have much more semantic information. And so for example, if you want to cast integers to years, what you can do is really just Call date object, open list, close list, close date object.

Yes, well, you notice that's the fourth thing. There are specific functions like.

So let's take a look. So there are four ways to change types, you say? Okay, so first point is this schema thing here. Okay, yes, this is, this is the thing where you're giving. Okay, so let's just understand how this, how this is supposed to work. So every column has a label, right?

Yep.

And normally in something like construct columns, you would be giving a thing like this where the right hand side is a function, is that correct?

That's right, yep.

Okay, so. You're saying that in a sense, you know, this is like tabular. Schema of a tabular is giving you a thing like this. Now, in a sense, Or maybe the right approach is. Well, okay, here you've got column types, which is the thing that. Is that an existing option?

Yes, there is a rough page for it.

Okay, that seems quite reasonable to me. But so now extracting the column types. So I don't think this is the name that you should be using there because I mean it should surely be T1 of column types because it's, it's basically it is returning a thing that you can use for the column types option.

Yep.

So, all right, now these types, how consistent with the compiler types are?

They're quite consistent. The one that is a little bit less consistent is date.

Because.

We'Re adding sort of metadata information about the date. If you look at the schema below where date has been converted to the full type, it has things like. So what's the grant? What's the grant? So it's the, the.

That's. Look, look, look. Just a ground rule. These types are going to have to be consistent with compiler types. So, you know, if the compiler doesn't yet support that, one day it will. And maybe for right now, today we're not going to do Julian dates in, you know, in. In tabular as a type.

Yep.

Right.

Yeah.

Because it's got to be consistent.

Well, the thing we can certainly support is the, the thing with the granularity, which is that second argument. And, and we can definitely support that in the compiler.

Okay, but, but what is the fully decorated form of this type specification, even for an integer? What? What? In. In the compiler there's a way of saying type or typed. Right. You know, you'd wrap typed of integer 64 and things like that.

Well, that's declaring that variable to be of.

I. I'm aware, I'm aware. So I just want to know what is the fully decorated form?

So the fully decorated form would be its date of the. That the integer there is the actual storage type. So that's the, that's how it's actually.

But maybe Christopher could say, even for an integer 8 in the case where we're saying, you know, euro string, delta arrow, integer 64, instead of just having a string there, what is the fully decorated form of that in compiler Type.

Specifier of quotes integer 64.

Okay, fine, yeah. And, and, and that type specifier has this. Let's just look at what it is when it has a, an array decorator. So that's the colon, colon, bracket, is that correct?

Yeah, that's just for parameterized types like date. So I think, I think. Right. Date, for example, might be a parameterized type.

I'm not sure. Tell me what the syntax would be there.

Well, yeah, go ahead, Rob A type specifier around the date.

Okay.

And then of the parameters.

Ok, so now. And there's a short form of that. Right?

Yeah. Colon, colon, bracket. But again, I'm not sure if date actually is parameterized. I mean, date would be parameterized if, for example, you wanted to have, you know. Oh, I know all my dates are going to be representable with integer 32s instead of integer 64s. I don't know why you'd know that, but.

Well, the parameterization, the ones that are to me are really important. I mean, I think generally we're going to have a Gregorian catalog and Nick and Jose would be able to talk about. But for example, that second argument is a granularity, and that's an important parameterization of it. So we're still storing it as an integer 64. We're interpreting those as having a granularity of say, nanoseconds or microseconds.

Okay, so something that's missing in this discussion is that in tabular, the dates are stored as UNIX times. And so that's where we say and the number of these unix times, an integer 64, an integer 32, et cetera.

Right, but.

And then the other parameters of dates are all there. So this has the exact same power of. This is super powerful.

Okay, sorry, hold on. There's two concerns, though. The first thing is, what if there's like inhomogeneous granularities, for example, or different calendars or something like that.

In tabular column, you need a homogeneous type, right? And so what we do is first parse through conform dates that will decide.

A common type, or you represent that as an expression or inert expression type.

Column, Right, right.

Okay.

Yeah.

I mean this is always the. I think that the challenge with a bunch of the types of things is just deciding which things are. Are you going to require be homogeneous and which things are you going to, you know, allow to vary.

Right.

So maybe with dates granularity is pretty easy or like, maybe calendar system is pretty easier. It's not often you're going to have a column with a mix of calendar systems. But I don't know, I mean, granularity is more on the edge or it's not as clear.

Maybe.

I mean, worst case, we could have multiple date types that are things like. I don't know what you'd call them, but it would basically be a date.

With an, you know, mystery and then one with a. Yeah, yeah.

I mean, maybe there's some syntax of writing that in the parameter. The type parameters too, I don't know.

But okay, but yeah, look, hold on, hold on, hold on. The thing we need to understand is in the end these are going to be done like the compiler, as type specifiers. All of the structure of the. Okay, so when you ask for the column types here, you will get back a thing that does not just have a string date, it will be a type specifier of. That's 64 or whatever, right?

That's correct.

Okay, and so we might want some type setting of the type specifier. Okay. So it doesn't look as like that. I think it'd be nice to have some box. It's not something relevant for the compiler because mostly the compiler is write only, so to speak in terms of the type specifications. Is that a true statement?

Yeah, although I think it would be. I think it would be good to have better typesetting for type specifiers anyway.

Okay, fine. So we need a design for the type specifier type setting.

So what I wanted to ask is in both interpreter and the databases internal type system, we distinguish between date, time, date, time and some of those can have time zone.

Okay, you're getting into, you're getting into micro details here.

If we have a different type for time from day.

No, I understand. I think we should have a variety of types.

There's a different time.

Right, but let me just say I think supporting more types, the real question is can the back end support these types? If it can, we should expose them. It's pretty much as simple as that. There's no point in distinguishing types that the backend can't support. But if it can support it, you know, expose it. It's cheap to have more types so long as we have a workflow where we can get the types from, you know, data from the wild. So let's, let's keep going here. So the main issue, so in this case the fully decorated form is type specifiers. I'd like to understand how the parameterized types work. There's clearly a sub discussion about parameterized types for dates. Okay, so this, you know, this way of saying it seems reasonable to me and this will then attempt to cast. My opinion is that what should happen is. Okay, so I think there should be another option here that says, you know, I don't know what it's called. Column. Why is this typing so slow? Something has gone wrong with typing. It's some autocomplete problem. Yeah, this needs to be looked at. Okay. I mean, this is 14:1 and it needs to be looked at. Can somebody make sure that that happens?

Yeah, I will check that.

I mean, it's like the 15th time I've reported the same.

Yeah, I mean, I make sure that.

This is really reported. Can you get our node to please actually look at this and find some measurement method for figuring out how typing is working?

Okay.

Okay. All right. Anyway, you know, I don't know column type. I this is bad naming, but mismatch function. I, I, I can't really type here. This is crazy. Yeah, it's typing unbelievably slowly. What is it? I mean, this is garbage. What the hell? Okay, okay.

I mean, for column types, though, I think, I think it would be good to have some like, you know, philosophy of, of what?

Like.

Okay, right. I mean, with associations, we had the associations or lists with named indices. Like if there's an analogous thing here of, you know, that, that basically a typed column always behaves the same as, as, you know, that, that, that whatever. If you have types that are compatible with the stuff you're storing, that it doesn't matter what type it is, it's just purely efficiency versus. Oh, it's also for sort of type checking for, to check if you make a mistake. Like.

I understand, I understand.

I actually added something for that very thing that I wanted to bring up. And, and so, so, so a thing which would be for efficiency is a bit like type hint. And so what I did is, and, and this may be incorrect as I interpreted type hint to possibly have an operator form.

Okay. I think type hint evaluates very eagerly. So it might.

No, but, but if you do type hint, if you think of the operator form type hint of unsigned integer 8. Okay. That if we think of that as an operator form type hint of unsigned integer 8 says try to do it on unsigned integer 8 but if it's not, put it in, put it in whatever type you can fit it in.

Well, that's an interesting concept I think type hint, at least in the compiler, if you say type hint of, you know, 12 comma string. Yeah, it disappears, just fails. I thought in the compiler it fails.

I could be wrong.

I'm pretty sure it fails though.

Is.

The same as typed in the compiler except that in the interpreter it evaluates away. Right.

Well, would in any case, whether it's type end or something else would be nice to have something like that. That's a, that's a guide but is not necessarily enforced.

Right, okay, but, but let's come back to some more basic things. Okay, so in this case where you've given, where data is a pure expression level object. Okay, and now you say I want these column types. And. What happens if they don't match? What happens if this is not doable as a date? We need to define this mismatch function. A bad name, I mean, but.

Right.

In other words, there's several possibilities. One possibility is everything that doesn't fit a date is a missing. Another possibility is you generate a message. Generate a missing and a message or something else. Making sense?

Yes, yes.

So, I mean, okay, so that's the thing we're going to need but let's just look through what it is we're talking about here. So we've got, with a tabular or two tabular, we've got a column type specification, right?

Yes.

Okay. Now in import, I think. Well, okay, if it's coming from a database, the database already has a schema. Am I making sense?

Well, but its schema might not perfectly align, but yes, right, but most of the types in it.

Yeah. Well, let's ask that specific question for the SQL instances that we support, is it the case that we can easily map the, the types?

It's a rather complicated question, Stephen, because what really happens is that effectively there are two different type systems. One is the generic one to which usually all this sort of functionality like what we have or what Java have, whatever, reduce the types which is used most of the time and the other is a specific type system for a specific backend that supports more types. And so depending on like in most cases we use this high level type system and that's what we also use for databases.

Okay, so any specific case like you know, postgres or something, having some fancy geo types, for instance, those types we.

Have to support separately. So.

Okay, but like Stephen, just to give you an example, like dates are a total zoo. They have different precisions, they have different origins, they're stored in different ways. So if you try to document it like a compiler type, trying to pretend that they are byte compatible between vendors, you're going to have like just a zoo of things. It doesn't.

Right. I think the point is not by. I think the point is not byte compatibility, it's just if we import from an SQLite database, can we represent it nicely in a tabular? Right, yes. Like, I mean, if they have, it doesn't matter if they're five different date formats, as long as we have a date format that can possibly represent what they're doing.

What I'm imagining is that just like for the languages, you know, like Python or whatever else, we are doing type conversions from their native types to our types. So when we bring online another database and we have a bunch of them already, you know, another database capability, we are, you know, that is, we are defining the conversion of its types to our types. Is that fair?

I guess that can potentially be achieved. But then the question is, are we only talking about conversion to in memory tabler, or are we talking about the things like, you know, querying and all sort of incorporating that conversion also in the query languages, et cetera?

I don't understand. Why would we have a conversion? We, we have. Look, for our purposes, when it comes into. I mean, it is useful to know, okay, there, there are two levels. What you're saying is in an out of core tabular, that there's actually two kinds of types. There's the truly native type, the external native type, and there's our interpretation of that type, which is relevant for our compiler to the native, you know, queries of that database system. Am I, am I right?

Yes. And also it's relevant to the semantics that we imply because essentially the types that exist on the backend, they may have specific semantics. And the question is, if we write queries like, for example, Memory Tableau has a query language that we are designing the query transformation language and we expect certain semantics implemented by this query language. But then the question is how faithful we want to be with that semantics when we use the same types, so to speak, in other backends, like whether the results that we get this way are the same or not essential.

I'm sorry, for what? Give me an example.

Okay, here is an example. For example, there is this decimal type in databases, which is not exactly our big integer, like arbitrary Precision integer.

I understand.

And so certain transformations that are in databases performed on that type of will be similar, except not quite the same as those similar to transformations we can expect from the arbitrary precision integer types. And then the question is, if we do the query on the other backend using decimal types, but we map them to the arbitrary precision integers, then should we expect what kind of semantics we really want? Wolfram language semantics or the semantics native to databases?

I think another way of saying this is if you make the same query by doing it lazily in the back end or by importing.

Exactly.

Okay, so let me give you an analogy. Back in the day when everybody wasn't the same endian and so on, you know, machine precision numbers would sometimes be different on different machines. Everybody survived. You know, there's a certain degree of approximation for these things that I think is acceptable.

Right. As long as we understand what sort of the compromises we're making and what expectations we have.

So we will not be able to get if we want efficient querying purely in the back end with across different databases that may have different conventions for different things, we're not going to get, you know, it's not going to be always compatible. So let's not try and make it that way. We're doing our best.

Right, right. So if we. Sorry, go ahead.

Yeah, What I was saying is that. But on the other hand, if it's not always going to be exactly compatible, what does it matter? Like maybe it would be useful to have some kind of a high level view, like interpreter types rather than compiler types, where you don't say integer 64 or 128, you just say integer.

Right. So basically what Carl is also saying, Stephen, is that there is no accident that various languages introduced intermediate high level types for basically intercommunicating with databases. Because when it's acceptable to do these generalizations, it kind of saves you from all this zoo.

Yeah, okay, all right, fine. Okay, so look, if we say, you know, such and such a column is integers, it's an interesting question, what does that natively actually turn into? And there's presumably some convention for that. But let's go through the high level design. Let's not get wound up in this. Okay? The high level design is there's a column types option for a bunch of things. There's a description on the way in of what to do if there's a mismatch in casting the thing to that column type. Am I making sense? I don't know what happens if you do an Operation. So this column type mismatch function, or whatever it's really called, will also have to be provided in transform columns. Because if you had a column that was supposed to be a such and such and you change it so that it no longer fits in a such and such, what is it going to do exactly?

I think in transform column, the contract is that even if it has the same name, the new column is a new column. So I don't think it's going to fail if you do the wrong operation there.

Well, okay, so it needs its own operations though, that.

But we may not. For example, if somebody does max on a string column, do we want to give a symbolic result?

No, I think that for something like this, when you do, there are also situations like you divide by zero or you overflow. In general, you put missing. That's the best thing to do.

I understand that, but that's exactly why I'm saying we need to parameterize that. Like, do you get a message? Do you not get a message?

What happens, Steven, one command here is that in database, in our Wolfram language to SQL query compiler, we have automatic casting inserted inside the database queries to actually conform to the semantics that we expect. So it's not just the level of the hints that you give when you just transform the column, but also the query language in that.

I understand, I got it. I understand. Yeah, okay, I understand that.

One thought I've had on this, Stephen, is that if somebody gives that column type mismatch function, or, you know what, whatever it is, or a set of, you know, an association that tells you how to deal with stuff, maybe that could be something that's stored in the tabular. And then when you do a transformation on that tabular respects that option you gave it.

I think that's reasonable. But let's, let's go through the other cases here. Okay, so another thing we might want to do is what you're saying here, which is to do an explicit casting of a column. So, so one thing is, okay, let's look at the workflows. One thing is on the way in, in CSV import, you're doing this thing of either identifying the columns or cut, casting them to a certain form. Okay, so one is on CSV import, right? Database import, it's less important because the database is really going to have quite granular types already specified, but the CSV will not. Am I making sense?

Yep, yep.

Okay, so then the other cases, it's coming purely from expressions. And again, you might have to cast things and then you might, okay, you might sort of precast them, like precast concrete. Precast them as it comes in from the CSV import. The other thing is you might post cast them. You've gotten it in, in some raw form and now you want to go and say, I want this specific structure. Does that make sense?

Yep.

And I mean, two tabular seems like it's a much better way to do that than something like this maybe. Now the problem with that is that then you have to say, I mean, to me, you know, it's like a transform columns. This, this failure to type is just driving me nuts. Can we get some kind of report about this in the next like day or so, please?

Tabular with a column types option makes some sense to me.

I agree. That's what, that's what we're proposing somewhere up here, right? But I'm, I'm suggesting that what we would do. So one possibility is that we say transform columns tabular, and then we have a column types option here that only specifies the columns that we're trying to re. Retypify, so to speak. Do you see what I'm saying? The problem is that you're not specifying all the column types. So one possibility is it's kind of like show and where you say tabular and you give some column types and those column types get appended to the column types that are already known. You see what I'm saying? It's the difference between transform columns and construct columns. So that's one possible strategy.

This seems to separate operations, right? So you might want to do a transformation and then a change of types, separate operations.

I don't know. I would claim that you rarely ever want to do a casting that it's not also performing something on the data. Like you can't. You have integer. You can't say cast date. It's meaningless unless you say, you know, you want to say unix time or julian date or something like that.

What about like the categorical case, though? Like, I've got a bunch of, you know, there's a column, that's the name of the state. I just want to say, just make it categorical for efficiency.

Good point, but I think that. I don't remember now exactly, but I think there were some functions that we were planning to do, stuff like that, like number categories.

We've got all the interpreter types. And how do those merge with essentially the compiler types? And do we imagine that one day there will be a compiler type for essentially every interpreter type?

No, I.

Wait, hold on.

Interpreter types that's ever going to happen because interpreter types are for interpretation. And so for example, we have a different type for string and for like a block of text text area. And the only difference for that is that if you enter a new line in the field, it doesn't send the form.

Yeah.

Okay, so there's some stuff, I mean there's also stuff like we have US state versus but it still comes back as an administrative division entity. Right?

Yeah.

Like interpreter types have nothing to do with how it's represented. They just have to do with how to turn it from an input into.

They have some with how they're represented. And actually if. I don't know if you can ever talk to you about this, but his original idea was to have an interpreter type and like an interpreter function and a serializer function which were like one the inverse of the other. And so I think that a type could be defined by two types in this sense. So you have one that serializes and one that deserializes.

But. Yeah, okay, but I mean that's relevant also for the storage and the raw underlying storage here. Okay, look, we have decisions that we have to make now in order to get this, you know. Okay, I just want to, I just want to take a quick look at this because I think it's relevant to what we're dealing with here. Okay, I'm not quite sure what that. Okay, should I look at this very quickly?

Yes. So there is a reference page called data connection. You bring that up sort from the sources. So the big picture is that we want a bunch of data connectors and the typical, I think analyst notebook data miners. Notebook starts with getting data at the top. And that's going to often be a data connection because data lives somewhere.

Yeah, yeah, yeah, yeah, yeah, yeah. Right. Obviously. Now, I mean that has to be merged with, you know, we've got resource data, which is our resource system. We've got the cloud, which is another source of data. We've got entities database. Yeah, we've got entities, we've got databases. Right, yeah.

So think databases fits in here. This is typically a query lang. You know, often a system has some kind of query language.

Yeah, okay, so. But there should be a way of wrapping a tabular rather than importing this into memory, which is not what we want. Right.

I mean for now we do want that.

Yeah, but we're not going to want that for very much longer.

Well, I, I mean, I think it depends that that's not a. I think this, this should just be a different import element Right. Like it's, it's. You'd want both behaviors.

Yes.

There are plenty of times when you just want to bring the whole database into memory.

Yeah, okay, but, but, but another thing that you might want to do is basically say from this data connection, create a tabular which has a remote backend. I mean, this is.

Stephen, I think, I mean, data connection is actually a very elegant generalization of what import does. So I think, you know, you have to view it in that way because it's something like. It's also a general, you know, just like the fact that import can work on a URL or on a file.

Right. Or a file. So it can work on a data connection. Right.

So the thing about data connection is right now.

Yeah.

Right now with S3, what you'd have to do is some sort of procedural workflow where you download the file, keep track of your temp files and all of that stuff. This takes care of all of that mess for you.

I am.

It's a data connection to some remote resource. Internal caching.

Also authentication. Authentication is.

I get it, I understand. But there's. It will very often be the case that there's a lot of stuff in this bucket and you'll want some. If I'm not mistaken, you'll want a slice of what's in there to be. To import. Or is that not true? Or is it just a pure. I want everything?

No, that's the point of objects is you could give an association of a name to which objects you want or just a list of which objects are important.

Well, where do we, where we, where are we doing that?

You see, take a look at the databases example, where there's a couple of queries. Right, because that objects example just has one string. But for databases you have queries and there's two different, you know, pieces of data that are saved in the metadata of this connection. So that's a connection to the SQLITE file.

I, I get it, I get it. So, but, but this is. Oh, I see. Yeeps.

But all the format of that is, you know, specific to SQLite. Right? Like in the documentation for the SQLite data connection type or whatever, it'll say you can give queries, and the queries have to be SQLITE queries. And in for S3 it'll say you can give objects and they have to be. Oh, yes.

So what's the capacity design with resource data? What's the compatibility of this design with resource data? Remember how resource data works? Does anybody here even know how resource data works?

I mean, in. In a way, resource data is a little bit like S3, isn't it?

Yeah, look at resource data. Okay? Resource data has this notion of an element here, okay? Right.

But resource data is.

Right.

Is, is, is. Is. Is eager, Right? It actually pulls the whole thing in, right? It doesn't.

Yes, that's correct. That's correct. But what it's doing is it's pulling the whole thing in. But just like this pulls the whole thing in as well with that sub value. Right?

Well, it pulls in that specific query, and the next time you run it.

It pulls the thing in just like this pulls in that element. Right? See what I'm saying? This pulls in that element of this resource.

Well, can we try evaluating that and see what happens outside of the import? I think it's the import that's pulling everything, not the query.

It's broken in this prototype build. It would work in today's build if there was one.

Okay, right.

So you're saying that's still symbolic, that query?

Yes.

So that query doesn't get executed the first time until you call import of it and the second time.

So the analogous thing in the resource system, which. Does anybody know what the analogous thing is in the resource system?

No.

Okay, you need to. This is what we need to do. We're not going to invent 75 new mechanisms.

Resource system is very specific to our data.

I mean, yes, it is somewhat.

You know, this is.

I understand that, but. So there's the notional resource object. Okay. The resource object is the kind of, you know, unevaluated version of this.

I think if we want to. I'm sure we can wrap resource object as another.

Just a second, just a second. Let's learn what we already did here and let's see how to apply it here. Okay, so resource object is a thing, like data connection here. What I'm not remembering myself is. Right, so I think. Right, so different resource objects. How does this work? Oh, this is a pathetically inadequate page. Somebody please put this on the list of pages that are pathetically inadequate. Anybody? Or am I doing it? Okay, so it goes to the documentation team for their list. Okay. Basic point is we got to look at how this design is done and compare it with this design because it's solving very similar problems.

I don't feel that that's true. I mean.

Yeah, I don't.

Abstraction from.

From. Right. From R is very specialized to getting.

I know, I know, I know. It has. It has mechanisms. Resource objects are not particularly specific. Resource function is specific, but it is just a sub casing of resource object.

Except you need data connection type and all those types are things that we provide.

No, you can define your own. It doesn't care. It just needs a compatible design. I'm not sure. Okay, so let's go start taking a look at this. So the concept is that there's a sub case of resource connection, of data connection rather, which is just the nounified form. Right? This. This thing here. I'm. I'm very confused by this. This has multiple objects in it. Am I correct? No, I don't.

This one just has one.

Okay, so then import of this. So if it had multiple objects, what would it import as?

Right now, nothing.

So that's no good.

I mean, come on, it's open to design. It's somewhat fair right now because I mean, one, we want to have the most absolutely simple workflow to just get your data and to go.

Okay, okay, fine. Look here, on the other hand, you've got these queries and this thing here stays unevaluated. Am I correct? On its own, Right, well, when you.

Run this, it will evaluate that to a different object.

No, no, no, no. The import evaluates it, but on its own, this thing here does not evaluate. Am I correct? Am I correct?

I don't know what you mean by on its own. Sorry.

Okay, why don't I try typing it in?

You're correct, Stephen. It does nothing. That's what Sean's saying.

Thank you. Does nothing. Okay, it is. Okay, so that is a. What is that? What is the input form of that?

So the association, the first association is kind of user facing details and the second association is internal data.

Okay, but I'm trying to understand how does it know it's listed all the queries. But this is using a specific query, whereas.

So there's a query goes to query one inside that first association.

Yeah, penultimate line.

That's really horrifying. So it's, it's all queries and then it says, after it's given all queries, it then gives a particular query. Right, right.

It's specifying which is the active query. Right.

That's extremely obscure. I mean, it says queries and then it says query.

Well, but I mean. Okay, how often are a user is really looking at this?

It doesn't really matter. This is. Okay, but this is a. Okay, if I tried to import the whole data connection, is it going to work, by the way? Is this going to work if I import this?

I need some design for that. It totally could work. I'm just not sure. What exactly that should do.

Okay. I mean it probably should return an association of the. But. Okay, so let's just look.

It depends a little bit on different. I mean. Right. I mean it works for. For SQLite. I'm not sure how it works with. It might be required different design for S3 or for whatever else. Right, right.

Well, the point of this is that it should be fairly intelligent what it does based on the type of data connection. Right. If you import a data connection of a general S3 resource, the most logical thing to me is list the stuff that's there. Kind of similar to directory.

How does this relate to local cache?

So redownload triggers getting the data from the remote resource. So it's going to use the local cache unless.

Okay, I understand that. How does it relate to the function local cache, which has the same issue. Right. Okay. So you need to look it up. Yep. Rather than just trying to make all this stuff up without looking at what we've already done.

Right. But local cache is for users to manually cache things.

I'm aware. I don't know how local cache is re triggered. That's what I'm asking for.

Oh, I see.

Because that's the same issue. That does not seem to say how local cache is re triggered. Which is kind of odd.

Well, presumably you can delete the local object and then it'll definitely have to cache it. But you're right.

I know, but I don't understand. I mean, it seems like it needs its own version of recache the thing, redownload. It seems like a thing that should work the same way.

Well, one, one, one workflow that we. What we think is very good is that if you refer to not local resources, but somewhere else, you can hand that notebook or that analyst task over to someone else and they can run it. Right. You can rehydrate the data and you're.

Able to obviously work obviously, if you've got the right authentications and so on and so on and so on. That's the same thing that we're doing with persistent values. Am I making sense? Which also should be, you know, should be interoperable with this. Right. You know about. For certain values.

Yes.

The other thing that data connection is doing is what is.

You do know about persistent values?

What is it used for? What is persistent value used for?

It's used all over the system for all kinds of things. For keeping preferences. For keeping all kinds of stuff.

It's across sessions. Oh.

Yeah.

In several different places.

Yeah.

I mean it's not used all over the system. It's used.

Yeah, I've never seen it used.

Well, it's used behind the scenes and lots of things.

A few things. Not lots of things.

It's pretty useful to store configurations of where I scan that should survive the session. Reload.

No, no, obviously. Very nice.

Like I use it in our link.

But just never seen it used.

For.

Example, in your analyst thing. Right. It allows you to do persist in the cloud so it's publicly accessible to people. Persist, so it's accessible to people with certain permissions and so on. Am I making sense? It seems very useful for this.

Maybe. I mean, I'm not sure we want to sort of pull that corporate data, upload it into our cloud and shit like that.

Yeah, I mean that presumably system credential is more relevant for.

Well, okay, this is really important. Can we, can we pull up the authentication page where Julian, Julia and I have made some updates? It's going to be in CVS because they needed this for chat and have already started implementing some of these things. That tentative box is my Julio and my new proposal for expanding on authentication to make things a bit more organic. And the key things here are that it's going to have system credential like you just mentioned, and also a secured authentication or an authentication dialogue.

So that.

You can just type it in and those things would be kind of saved as system credential.

Seems like a no brainer. I agree with that.

And notice that I changed a couple of things like for automatic, instead of making it so specific to cloud connect, I say default credentials.

Yes, that seems reasonable. Now what's this here?

Authentication dialog is pretty cool. You could pull up the page.

No, I know what authentication dialog is. I'm just trying to understand that the option.

I think it's. I think it's. When you run it, it will bring up a dialogue which seems.

I know, but why is it running? Why is it given here with no arguments, so to speak? So you see what I'm saying?

Because behind the scenes it saves it as a system credential, essentially. So we keep track of that for you.

No, I'm saying.

Authentication dialogue.

Right.

It needs to create a unique dialogue for this particular thing you're trying to authenticate for. Right?

Yes.

I can't say authentication dialogue of foo. Right.

No, I understand. It's just very weird that this is a pure symbol, whereas all these other things here are sort of complete objects.

Yes, because those contain the, the. All the data. And this is going to generate something for you.

Right.

Automatic is also a pure symbol.

Right.

So this is a sort of automatic authentication dialog.

But isn't it okay in this case of system credential, that's just a key effectively to the authentication system to say user system credential. But here, if I want to decorate the system, the authentication dialog that I.

Am using, I can't give it a specific key.

Not a. Not a generated key. Is that what you're saying? So like.

Well, I'm just saying let me make it purple and. And have all kinds of, you know, decoration and so on, and say, now you enter the authentication and so on. Because just giving authentication dialogue as a single symbol, you have no choice about what the dialog is going to look like.

Well, you don't, because we construct the dialog for a particular type of connection.

Right.

S3 is going to look different than the dialogue for Dropbox or a database.

Right.

But if you wanted to. I mean, I think this is a pretty marginal point. I think the point he's trying to make is if you wanted to make that S3 dialog purple, like if you wanted to customize that S3 dialog in addition to all the automated things that have already been put into it. But it's pretty marginal, I think, like, most of the time. Well, yeah, I mean, most of the time. First of all, I'd imagine this dialogue is being used by the person writing it. I mean, in the cases where it's a third, you know, like, I think the main application of this is like, I'm running LM Synthesize and I just want. I don't want to have to actually type in some API key. I just want a dialogue that will handle all the interface that I don't have to do it.

Okay. So it actually initiates the dialogue.

Yeah.

By the way, I actually have to run right now, but.

Bye, everyone too. But.

So another thing that I wanted to bring up, I think.

No, no, no, no, no, no. Wait, wait. This thing here isn't right. This design isn't right.

I mean, if.

Right.

If you want to call it like string dialogue and then it's. I mean, are you complaining about the fact that it's a symbol or the fact that you can't customize it?

Because I'm mostly complaining about the fact that it's a symbol that has a definite meaning. And this isn't sort of an abuse of that symbol?

Yeah, that's definitely abuse. Yeah.

I mean, we could call this string dialogue, and that would be a magic.

String that means call a. I think that's reasonable. I think that's the right thing to do first.

Yeah, that's, yeah, we never do this symbol and option name the same, just never.

So, okay.

Value, not an option name. Which is.

Yeah, I know it's a little different, but so I think initially just call it quotes string dial. You know the string dialog. And it does, it will do, it will use behind the scenes. It will happen to use authentication dialogue. Okay. Okay, we need to move through some things super quickly here. Let's just enumerate what our issues are. Issue number one in data connection. You know, I understand the, the sort of one gulp data connection where there's a single thing that you can get from the data connection. Okay, I do not understand the thing where there are multiple named gulps. Or for example, where you are explicitly giving the query against the data connection. Am I making sense? Because if you can do this, I.

Don'T understand the multi gulp either. Okay, but we can make meaning out of it. The query. I do understand almost all big data stores. You need to say what it is you want.

I, I, I'm aware, I'm aware. So I mean the, the fact is the data connection is merely the authenticated thing that is like the what we're calling database object. If that's what, what are, what is the thing?

No, no, no, no, Stephen, I don't think that is.

No, no, just tell me what the thing is called.

Relational connection.

Relational database. Okay, yeah, that's the thing that represents. No, wait, no, no.

The raw connection is a database reference.

Okay.

Like just the connection relational database has pulled in all the metadata.

Okay, fine. So again here the analogous thing would be just driving me crazy.

Maybe it's not as much what we do with the import of the gulp of multiple connections, but if you connect to something like a database. Notice that that line of tables, that's something that you can do with a connection as a property that isn't just gulping everything. But I have this connection to that database.

I get it, I get it. So, I mean, this is what we've got is database reference is our way of doing this elsewhere. This is driving me crazy. I cannot use this. This is typing like it's 1976 or something.

If you try turning off, if you.

Don'T mind, you could try in the preferences to set a non zero delay for code assist and see if that helps.

You can also turn off fuzzy matching. That also might help.

Stefan, you need to track this down and get it fixed because this is not working.

There is an Open bug about, I think exactly this issue.

But it's not just an open bug. It say somebody needs to go look at it now.

Yeah, I, I pinged John FS about it a few days ago and he said that it's on his list for 141.

See what happens now, it's much faster. Okay, is that going to affect my version 14? Is that, does that, Is it shared preferences between this and that?

Well, that, that depends on how you have your system configured.

Okay, all right, fine. So we have no idea. So could somebody please.

I'll, I'll be, I'll ping John once this meeting ends and I can actually go to my 3:30 meeting with him.

Okay. All right. Okay. Where was I? So we've got, you know, really, we've got a data connection reference which is the first part of this. Not all of this, you know, sub casing stuff. See, what you've got here is some standardized sub casing which is very similar to what happens in Resource Object. Am I making sense? Resource Object also has that kind of possibility of standardized sub casing. Anybody understand what I'm talking about?

I don't.

Okay, well, go study resource objects. Just go look how they work.

I thought it was a rhetorical question because you wanted to explain.

Sorry, no, I don't want to explain it. The. Okay, so look, roughly, what's got to happen here, I think is there are two levels of this. One is the raw authenticated connection. Okay. Essentially it's not quite, it's not quite data connection reference, but it's something like that. And the second thing is just like relational database is the wrapped thing that you can then ask for properties like one of the tables and so on. I think it's the same thing here. Does that make sense to people?

No, I mean, we wanted to make it much simpler than that.

Okay, Roger, Roger, we will make it simpler after we figured out what the correct semantics is. Okay, Then we will make it so there's a quick path, but first we're going to figure out what the correct semantics is.

Sure, we can do that. So the other thing, if I can bring that up now as a wrapper, is that I think it's a great opportunity to build a data connector repository or a connector repository.

No, I think it's a very good.

Idea because we need to many classes of these not only databases. There are APIs. There are.

Well, we've already got a mechanism for APIs, don't we? You know how it works, don't you?

But that's unusable so what do you mean it's unusable?

What the hell are you talking about?

Service Connect.

What are you talking about People all the time.

Yeah, yeah. That's not.

People use it all the time. The basis for tons of stuff. We do.

Yes.

Oh, I think that we can make those API connections even more elegant with Data connection. They can be. That would be nice. And give a more consistent and elegant interface to them, you know.

Yeah.

Service Connect will still be there for that.

Sometimes you can, sometimes you can't.

Exactly. And for the ones that we can, we want to promote this to data connection.

Right. So sometimes you can. And you know, some of these are nice, elegant things and some of them are not. Yeah, we.

We were planning to use as much of this and add more. This is a reason to expand on this. But it's not really.

It's not user. So we need. Yes, it is the thing for the ones that. It depends on how garbagey the outside API is. Some of these are more usable than others and some of them are the basis for things like LLM, synthesize and so on. Am I making sense? Right. This is the underlying infrastructure for that. We have a higher level wrapper. But it's not the right thing to just say we're going to have some hacked thing. You know, somehow.

We never said we're going to have something hacked. We said we're going to have something more elegant that's user presentable.

Okay, but that's different. You're going to build on this.

Yes, we are.

Okay.

We have been. That was the intent to leverage it.

All right, so fine. So let's just get. Okay, I don't think this is quite right. I think it needs to be two levels. I think it needs to be something that represents the raw connection and something that represents the kind of the presented things you can do with that connection. Am I making sense, Stephen?

I think that one way to see this is more like. And this is what I like about this, but if you think about it like there's a data connection to Dropbox, which is a general thing, and the elements are like paths to various files in it. Or if it's S3, it's buckets. And then you drill down and then at the end of the leaves of this, you find that the data. That the elements of that are file data, of course. So they're all their data connections all the way down. And I think like there is a data connection to a whole database whose properties are like the tables, maybe some stored queries.

I'm gonna have to go. So we've got basically just a couple of minutes here. Okay. We've got several things. You know, we need more time and we need the. I don't know when we're next scheduling one of these meetings Monday. Okay. All right.

Did we just pick it up there?

I think we should pick it up there. Look, we've got two things here. Could you guys please try to think about how to make. I mean, my, my gut feeling is that we're going to need two levels here, the raw connection and the dressed thing. And also make sure that we have some alignment with, you know, the persistent value, business resource objects, service connections and so on. So we have some consistency between these things. So. Yes, I understand, Roger. It's much easier to just say all the other stuff is garbage. We're going to do something new. That's not how we do it.

No, I wouldn't say that. But if we, if we feel that we can't change a damn thing in the system, then it will never evolve.

You and I have been doing this for decades. I know how to do this. I know how to do it.

I know.

Come up with what we want to do.

I know.

Not ignore what was there before. Figure out how to bridge the whole thing. But we don't ignore what was there before and just say, oh, we're just going to do something new without thinking about what was there before.

And it's built on top of those things. We're absolutely not abandoning it.

Okay, fine. Then make sure that the design is compatible. We will been designed. And the design will be different as well. The design will be compatible, but different.

Yeah.

You know, as it always is.

Right. So this is the first time you see it.

Okay, let's come back to these, these type things. Okay, let's just, I mean, let's just talk for a second about that. So as I understand it, there are, you know, there's how you bring the thing in, how you cast from external types to our types, but it's a little bit more complicated because we may need to be constantly dealing with the external types because we are dealing with an external. Because we're dealing with external. What am I saying? You know, we're dealing with a. With an external repository for the data. An externalized data connection, basically. Am I making sense.

Tim?

Yes.

Although, I mean, there's also. Sometimes you just want to convert between types internally because, you know, you know, you want to format it.

I understand. You want to cast it in some way.

Yeah, yeah.

So I mean to me.

And that's. It's almost part of an analysis or data cleaning. It can be part of both or either. Okay. Can be a cleaning operation that makes subsequent operation easier and more reliable.

It can be. Okay, fine, fine, fine. But just thinking about the design. Okay. There's stuff on the way in coming from the wild. To our types. Yep. To our types.

Yep.

Okay. Then there's. Okay, so. So basically there's a data connection which has certain raw types and then we have certain types that we know about and there's some correspondence between those things. Does that make sense? And so if it is going back and forth doing operations in this external back end, it is being converted for storage in our tabular. I don't know how this quite works.

That's the beauty of the arrow types. The point of arrow is that it makes it easy to pass data between languages. It was originally created to.

I don't need the speech. I don't need the speech. We don't have time. Just get to the point.

If there are arrow types, they're extremely compatible with other languages already and those are going to be the most common types.

Okay, and are they compatible with our compiler types?

Pretty much.

Pretty much that. I base the whole type system for tabular column on arrow types.

Okay. All right. Well then we also have got to deal with these enums and things like this. All right, we have to wrap here. We will continue on on Monday. And okay, there are a few things that it'd be really good to have a think about before then. I mean, this is in another hour and a half. Will probably have ground this down. I will see you all then.

Okay.

Sam.