Live CEOing Ep 800: Language Design in the Wolfram Language [Tabular]

Okay.

Hi everyone. Welcome to another Wolfram Language design review. I am told this is the 800th episode of this series. Wow. Good. Okay. Yeehaw. And actually we have three live streams coming up today. First we are talking about tabular, then we'll be talking about calculus and algebra, and then we'll be talking about biomolecules. Okay. But here we're talking about tabular.

So we have this section on types that we started on Thursday.

So we're not calling this join across is being called column combine. Right. Or column wise combine. Right?

Yes. So that was the discussion of modifying join across.

Okay, so this is all done. Yeah. Columnize combined there fundamentally.

Yes. We want to revisit tabular training with Julia and data connection. Of course, that will continue in hopefully soon. Okay, so today, the new section today is at the end. It sets operations, something that Carlo and Leonid can describe in more detail. And perhaps you wanted to continue discussing types, which we started on last. On the last meeting.

Right. So is this a cleaner list than we had before? Is this the same as the list we had before?

It's the same, but I think it's complete, it's completed. Right, Rob?

Yeah.

I mean, there's only a couple of questions in there. Like, should we call the list thing list factor? Because that's more like the compiler.

Well, is it a list or.

Well, it, it represents a list. So each element is a list.

Okay, I understand.

That's what it represents.

I mean, this is all pretty low level stuff. What, what is a list? Tuple.

That's, that's something that comes from the compiler. That's, that allows different types. So it's a list, but it has different types across the elements of the list.

Right. So it's like homogeneous resin, homogeneous list.

Well, Christopher, what would you call these things? I mean, this is, this is low level compiler land, this whole thing.

Yeah, I mean, almost all of these, I mean. Right. All the ones that have compiler equivalents, with maybe one exception I see, have the same name as in the compiler. And the other ones have seemingly kind of reasonable names. The only one that's a little different is in the compiler there's listtuple and list vector. But here there's listtuple and just list.

Let's call it List vector. If it's called list vector in the compiler, let's call it list vector here.

I'm happy to change it to list vector.

Okay.

And.

And binary versus byte array.

Well, what is it on the compiler?

It's byte array in the compiler.

Then let's change it.

Let's change it. Okay, fine.

Isn't binary just true? False. No, that's boolean.

No, that's boolean.

Okay, so.

So.

Yeah, this boolean the same as, as binary, or can it be converted easily?

No, no, no, no.

Byte array is a much clearer name.

Yeah, yeah, I could see, yeah.

The confusion that's being generated right now is why we should change this.

Yep, exactly. So I, I'm. I will change it soon.

Sorry, sorry, I was muted, so. Right, so when it says fixed width, it means fixed width in bytes. It's not a bit vector, it's a, It's a byte vector that we're talking about here.

Right, exactly.

Actually, what about.

Go ahead.

Just a quick question. So for something like that is the. I assume, can all of these be ragged or some of them? Like every row is to have the same length.

The list, the list vector, and the binary or the byte array can be. Can be ragged. Yes. Okay, so the, the one with the length qualifier says that it can't be ratted. They have to be consistent lengths. I see. Yeah, we can survive without the, the, the length one. It's sort of a memory optimization. But.

But again, these are the types for elements in a column or these are the types for a whole column.

There are types for elements in a column.

Okay.

What's the difference?

Well, because. Because we're not taking the whole vector of stuff in the column. This is just the individual.

Oh, yeah, right. It's a vector of this.

Yeah, right, right, exactly. Exactly. So that's, that's the example I gave below. The compiler type would actually be tight specifier of tabular column integer 8.

Right.

Well, why is this in quotes? Those things aren't in quotes. Right. In the compiler, which things integer when you write a type?

Yeah, those are always in quotes in the compiler.

Yeah, okay. Okay, fine.

Yeah. And sparse array and numeric array, they're in quotes too.

Yeah.

Even though they're heads.

Okay. Otherwise, nominal scale and ordinal scale, are those the right names? That seems completely wrong. What's that?

I just, I would argue.

No, I think it should be enum, basically, because that's what they are.

No, but there shouldn't be scales. I mean, I, I think. I mean, it's. In some. From certain perspective, calling them scales is correct, but I don't think in a type system, that doesn't seem right.

So what are they? Are they enums or are they not enums?

Well, they're. I mean, if you think of like super old programming Languages. Yeah, they would be enums.

Well, that's like the same super old programming languages that have integer32.

But the enums, I feel like are also a little bit different. Like enums. Like, for example, people generally assume that enums. That there's a, you know, an integer representation that you can access or something like that. Which is.

Which is not true of modern C, but that is true of. Of C and also older versions of C. Or did they call it. Maybe they called it something else.

Right.

C definitely has an enum type which is not castable to an integer. And maybe it was enum something. Maybe they invented a new name for that. He would know the answer. What's that?

Yeah, I mean, enum is just a very esoteric weird like. Okay, what should we call it? That.

I mean, the. A common, you know, a common name for Enum is record. I don't know that that still conveys the right.

No, no, no, that does more record.

I mean, what I would prefer also.

They are really different, the two. I mean, one shouldn't conflate the two.

Because there's nominal and ordinal.

Yeah.

I mean, we could have a blah and order the blah and.

Yes, that's what I'm thinking about. What's blah? Yes.

Maybe just category.

Enumeration.

Right.

I like the name.

What?

I like name categorical as an adjective.

What about enumeration?

Say it again.

Enumeration.

Yeah.

And here's another name for you guys which you reject, but in an article factor.

What? Yeah, because.

No, I want to understand why they.

I know, I was totally baffled. Okay, so here's the reason. If you build a linear regression model and you're going to use a categorical variable, each value is basically going to be a new numerical variable factor. So that's a factor.

Brilliant. Okay, let's just remove that. Okay. Carlo is suggesting choices, which I don't think is the worst thing in the world.

I don't love it. But it's what we use for the interpreter type system.

It's a discrete set of choices. I mean, it, it goes along with random choice. What is used in the. In the. I mean, categorical data, right. Is categorical distribution and so on. Right.

Given that we did not have not used collection elsewhere, we could. We could go with collection in order.

But it's not a collection. This is not, you know, if it's a collection, you expect that like it's going to contain multiple elements. But no, it's just one.

Exactly.

Thing.

Oh, I see.

But it, but it. But it does contain.

I mean.

No, it does. It doesn't. Well, individual thing.

The categories. The categories themselves are unique, but the. The whole vector itself may contain multiple instances.

You mean like. Like if I have something that's of type collection, I expect it's going to be like a set or something like that, but here it's just like a valid.

Okay, yeah.

That's not horrible. I think that's the least bad one we've come up with thus far.

What about.

I know this may be way worse, but selection.

Right.

I don't like it. That has so many other cursors and all kinds of things like that.

Yeah.

What is the point of the element part? Aren't these all elements?

So the reason you want to invent this name now is for the compiler or that it fits better with the compiler or.

No, just that it's a name that is absolutely incomprehensible. Every time I see this in this list, it is incomprehensible.

Yeah.

That's why I want to invent a new name. Okay. A nominal scale is something to do with graphics or whatever else, or, you know, or it's not storage, you know.

Also, I think half people don't know what ordinal means.

Well, if you're in statistics, you do, but.

Well, all right. Okay, let's come back to this, please.

Can I. Can I put my vote against using element in there? Because then people might think that it's the same as repeating element and compound element.

Well, yes, I don't think element is great.

Although it won't be a symbol, it'll just be a string, so it's slightly less. But I also mean the element does just add nothing. They're all elements.

Yeah.

Integer element.

By the way, in the boxes of categorical distribution, they're called categories.

So. Yeah, I was just gonna say maybe we should call it category.

Oh, gosh, no. Because we've got all kinds of category theory stuff coming along.

I see, I was not aware of that.

Well, I like categorical and ordered categorical. I think that's not bad.

Yeah. The problem with categorical is it's an adjective.

As in. I know, the battles in the order. Categorical.

What the.

It's the. The major general song.

Oh, yeah, wow. That's that older word, the categorical imperative.

That's an adjective.

Well, but that's not. I don't think that having an adjective is that terrible because remember, the. The underlying data and the normal stays the same. You're just changing the representation to be categorical.

That Is a. Is a recovery argument if ever I heard one.

Sorry, Stephen.

Okay. Well, I mean, certainly if one saw this, one would understand what it was. I think, By the way, even the.

Description, you know, we could probably do. Rob, you tell me. Let's say that we had categorical and ordered or not ordered is basically like a boolean in there.

Kind of like that's exactly how arrow represents it. And essentially that's. That's how I represent it in neo art, our reference thing.

So, I mean, in principle, boolean is categorical too, but it's, you know.

Yeah, I like categorical and I think, I don't see why we shouldn't have ordered categorical as well. I think putting it in as a. You know, because in these other cases.

Well, the only thing about ordered categorical is that it kind of sounded like at least last time, like, Roger, you were talking about grand plans for having ordered categorical things elsewhere. And so if we hadn't ordered. If we had something called order categorical here that doesn't do all that stuff, then it might be hard to add it later.

Well, I don't know about the names to. I mean, these are just strings. It doesn't matter that much.

Well, yes it does, because we don't want a symbol that somehow. What is the thing you're imagining is the later version of this.

So I imagine doing. And I, I've done it just rough to. To have them live on as a type of compound symbols in the, in the. On their own instead of just in a column or in. Just as a visualization function. Because then you can have lists of them and they know enough about themselves and how they order and so on. They can display nice. And it would just fit the language better in terms of how we explain that.

Again, what you have in mind, I didn't understand that.

Okay, so, you know, normally I think of date as a date vector, but we have these isolated dates in the system and they are an actual object and they have all this extra crud around them to deal with all that. But really, it's just like if you, if you get UNIX time, that's all that's needed, right?

That's all that's needed for humans. That's all that's needed for. Okay, but in a case. So I mean, your point is that, you know, you.

So right now, if you have just a string in the system, it's a little bit like UNIX time, you know? I don't know.

Yeah, I understand. I understand what you're saying. You're Saying that you want something where it is. I have an A. It is in the collection of, you know, a set of things, A, B, C, etcetera, Et cetera, et cetera. It's kind of like.

Yeah, and what we can do then is also curate and almost build a repository, if we like, of these categorical scales, particularly the ordered scales.

We talked about this years ago. Right, yeah.

And so there's a ton of them, you know, that's used in psychology. And there are these hardness scales, for instance, you know, the borderlines of physics, but in particular medical and.

Yeah, some of them are, Some of them are discrete. The hardness scale is not discrete. Right, right.

But, but the question is, so if we introduce this, then, for example, would indexing into an ordered categorical column, would that return something with a different head from what we have to do right now?

Yes.

Right. So that I would argue is an argument for not exposing this immediately.

Yeah, I, I agree with Christopher.

Which. The ordered categorical thing.

Yeah, just, just that ordered cat.

I mean.

Right. Unordered categorical is just pure optimization. And so it should just, you know, why not. But that ordered categorical sounds like it's going to mean something semantically different from anything we can implement right now.

Right, but it is actually kind of important in data science too.

Like we all understand, but do we need it right now?

Well, it sounds like the only effect it would have right now is stuff like if you sort on an ordered categorical column, then it can do something slightly nicer. Yeah, but frankly, I think it's like kind of marginal if it means that we're going to have to do some big breaking change later. Oh, no.

But also, like in this, you kind of want them in the, in the, in the right order.

Look, look, guys, I think it is sufficiently defensive if we call it categorical and categorical ordered, which is okay, given that we have real 32 and all those kinds of things. And you know, when we have a better design, the things which are the symbolic, you know, like a symbolic date, for example, whatever that name is, we can use that as another type in this collection. Does that seem reasonable? I mean, whatever this is, the categorical ordered thing will be some internal type that will have to be converted later.

Yeah, there's one, one thing I was wondering with respect to this. I mean, if, if I am asking whether an order categorical is a type of categorical, does that return true? Like are. Does one contain the other?

I don't know. I mean, does real 32 contain, you know, does real 64 contain real 32? No, I don't think so. They're just, just disjoint types.

Well, it depends what you mean by contain. Right. I mean, like, like, I mean, first of all, I'm not sure the tabula.

Right, right. You can, right, you can losslessly cast real 32 to real 64, but not conversely.

Right, but first of all, does, does tabular. Does the tabular system ever care about a concept of, of what types can represent what other types?

Well, I, I think, I mean, in general, I. And here I'm trying to introduce a new discussion item which is.

We need to move on though.

So that's, that's, that's my tentative moving on here. Like, I think that there are a number of cases where we don't want users to have to specify whether it's an integer 64 or 32, but they probably will just want to say integer and we do the right thing.

So.

And I think this is not just for tabular, maybe it's also in general for the compiler that to have the idea of this kind of super types that, you know, if they have to, like, if you have to do type matching, they match all of the subtypes. But if you have to do something like casting, it will choose something reasonable based, say on the fact that your computer is 64 bits.

Right. So the closest analog in the compiler, which is not really what you're talking about, though, Carlo, is there. There's abstract types. So you have. There's an abstract type called integers, and you know, all of the integer 32, integer 8, et cetera, are all members of that abstract, they're all instances of that abstract type. But there's none of that right now. There isn't any of this sort of automatic, like you just say integer and it figures out which type you want.

But the way I see it is not abstract types. I see it as like in terms of inheritance, really. Well, yes, abstract types, Integer is an abstract type, but it doesn't have to be the case that. I mean, in inheritance you can inherit also from a concrete type, not just from an abstract type. Like, that's what I meant with this categorical. Like one could be a subtype of the other.

Right. I think. Well, yeah, I think. I know Tom I think has some opinions about inheriting from concrete types and stuff. I honestly don't know. Right. But I'm not sure that we necessarily even need to have a whole theory of that to have something like just integer work and tabular.

Yeah, I don't know I mean for me really the motivation behind this is that if the level at which we document is integer 64, then I@ some point have to go and find out what postgres means by bigint and what it means by date and what the. Exactly.

That's what's going to have to happen. Yes, that's the work we have to do.

You mean as in when you import, you know, if you want to import from a SQL database and you at some point maybe need to tell it the type of the column you need to look up, you know, what's the mapping between SQL's types and R types?

Yeah, but like I don't think like there's a zoo of stuff out there and like different version of postgres to different things. I don't think we should be in the business of redocumenting what they did. We should just return the right thing. But I don't think that when you query the schema of an external database we should give you like an explanation of whatever they had. Remember we had this discussion about relational database back in few years ago? Like do we want to explain we.

Return the generic size version of that type? That's what we return, right? Correct. Right now?

Yeah, right now we do that.

Yeah.

Okay, so fine.

So the same. Okay, so we just need in this list a genericized version integer, unsigned integer, presumably real. What the heck is complex real?

I wondered about that.

What the heck is that?

It's specifying the size. That's what the compiler does. Instead of, instead of complex 64 and complex 128 which were potentially ambiguous. It's saying it's a complex based Unreal 32 and a complex space. They mean Unreal 64.

Okay.

We have something called machine integer and machine real and machine complex, for example. And that's, that's. So when you get an integer in Mathematica as a machine integer, it's, you know, that's what it is.

What do we need to figure out here? Because we've got a lot to cover today. What do we need to figure out here? Do we need, we need these generic size types. Maybe we need the machine integer type, etc. Etc. Etc. But what else do we have to cover here? Nothing. Okay, let's move on. Yeah, we've got categorical and categorical order to what we call is date in there and time data here and time.

That's not. That should be included in that list, but it's not.

And so that's a bug. So fix it.

Yeah, because in in the interpreter type system, date means up to date. It doesn't have the hour, minutes and seconds.

Yeah, our date is what's often called date time in our system.

Yeah, so maybe we should call it date time, not date, because we already have a precedent for that in the language fine interpreter.

Yeah, we support that name too, as an alias to general date.

Yeah, but date doesn't have the time.

Yeah, well.

Well, we have date object, which is.

You can have dates with granularity of date. You put the granularity you want and that's it.

Yeah.

Okay, so what are we doing here? Is this called date or date time?

I think it should be called date in general. Date time is a form of date which admits both date and which has both date and time. And then we have.

I don't know, but I think, I mean here we gotta move on. Yeah, but guys, there is a problem here. Date time is a timestamp. It's something that always has all the ground is not a calendar date. It doesn't mean.

Then what we've got here with date is. Is a thing that maps our date object and that's what we want.

Yeah, and I think that's fine.

Do separately.

What I'm saying is that daytime should always be instant.

Okay, well that's irrelevant.

Yeah, it is.

That's how daytime is defined.

Okay, great.

All right. Okay, so I think we went over all of the stuff. Did we not? This is. Yeah, yeah, yeah. Are we in fact having cost as a top level thing or not?

I think it should be there given that the command exists and it feels natural in transform columns.

Yeah, it's a little odd because it's. I mean right now that only. Yeah, I mean, I guess, why not? But it seems like it would be.

Good evaluate or not.

It does not. It gives an error that says it only works in the compiler.

Yeah, well obviously that's got to be removed, right?

Well, at least changed.

Right. I mean, Right. It's inside of function which is all. All complete. Right. So don't need to worry about it Evaluating.

That raises the question of should cost. Just evaluate. I remember we talked about this when we were discussing it with the compiler.

Right.

And I think we decided no, because there are some situations where CAST is doing like non trivial things. Like, oh, I'm casting from a dynamic array to a list vector. Those have different expression representations. So it's not a no op in the evaluator.

Well, I'm not expecting it to be a no op. And if you cast from a real to an integer, you're going to get some kind of weird bit pattern garbage.

Well, yeah, I mean, I think, I think it's just that, right. I mean, I guess we could like. Okay, basically the point is that the. There's a complicated cast primarily as it exists in the compiler, is for converting between compiler types. Compiler types have an imperfect matching to interpreter types. There are lots of things that have different representations in the compiler and are the same in the interpreter and vice versa. And so then we could make a whole nother version of CAST in the interpreter that would map between interpreter objects. But that's like. It would be many levels removed from what's going on in the compiler. And frankly, I think it's much better just to have functions, you know, convert from X to Y.

Listen, the issue that I have is if I'm trying to debug something and cast only evaluates in some weird situation inside, you know, transform columns, I'm going to be very confused.

And also I think that that example you have on the screen is kind of wrong because. Well, it shouldn't. Like you can't cast a year. You should be casting the UNIX time to date.

Well.

I think that's a different issue, right? I mean, year is the name of the column here, right? And we have an interpretation of such things, right? If we only have the year, that means, you know, midnight on January 1st.

No, no, no, but what I'm saying.

What if it is the year of timestamp? Like, is the year going to say 1996 and cast of date returns?

It's.

It's not going to interpret 1996 as a, as a UNIX timestamp or something, or an absolute time.

I think that if internally, if internally dates are represented using as. As UNIX times casting from number to date.

But that's not enough numbers.

Okay, guys, I think this casting thing is dangerous. And do we really need it other than as an internal.

No, we can, we can do that. We can do the tabular column I show above, but we don't. We need. We need the way to insert tabular columns into tabular.

No, but wait, why do you need the tabular column? That would be. If you want to do that cast year that is basically just date object.

Okay, let's see what we can do without having the explicit cast in here, because I don't understand in this case here, we want to say this is a tabular column that we are declaring to be of type date, correct? Yep. Okay, that seems fine. But we don't need this cast operation which who knows what that's going to do? I think you can normal the tabular column and then run tabular column again with a second argument, which is the, the specification.

Well, I mean you don't even need nor, I mean if we're going to do nested tabular column, we don't need the normal. Right. We don't need a column of tabular column.

But what we do need.

But, but I think Rob's. Go ahead, Rob.

What we do need is a way to put tabular column into a, into a tabular. Whether that's two tabular or something else. We were discussing that like two meetings ago, so.

But, but is it didn't. So your cast is still a way to, to change.

Convert. Convert the representation in place.

In place that we would like to have.

Yeah. Okay, fine. You know what? We are running out of time to do complicated designs. I don't think this design is clean enough. I think we should leave it out for now. And you know this. Let's not worry about in place type conversion. We worry about type conversion on the way in and so on. Now the question, the question that you're raising is if I want to insert a tabular column into a tabular, how do I do it? Right. Let's remind ourselves about these things here. How do I insert. How do I. Yeah, how do I insert a column wholesale?

Can, can we not do it as an L Value? Right. Well, actually, I don't remember where we ended up.

Lvalue is very risky if we, we have rows as the first thing.

Right.

We don't want people to try and change rows. That's expensive.

So I'm not keen on the, on the L value stuff. I mean that, you know, one way to do it presumably is with column wise. How would we do that?

Right. But you will need to join on row IDs for that.

I think if you look.

How do you not join? What does it mean if you're not joining on row IDs? I don't understand if, if you've got a tabular column that is coming in from the wild and its row IDs don't agree with the row IDs of things that are already there. It's kind of a, A toasted column. Sure.

Does the tabular column know its name? I thought it was just like here I'm a column.

No, I don't think it knows its name. It doesn't know its name, but it does know its row id. It does know its row information.

Right. Can maybe we discussed this and rejected this. But can we not do something like you know, transform column and just say, you know, in the special case foo goes to tabular column rather than using that as a function, it tries to replace the column with that.

I don't think that's unreasonable.

Oh, that's.

That's what works now. But one reason I don't like it.

Yeah. One reason we were against this was that transform columns in all other circumstances work on element like on row wise basis and having adding another special case where it works. And column wise basis to us is not.

Well, okay, so I don't understand why isn't column wise combined the right thing?

It is, but we. When we were discussing row or joining on row. It is. I think that part was a little bit left out because the we. We didn't yet decide how to introduce row IDs in a consistent pattern.

No, I don't understand.

So I could maybe you could just write.

Yeah.

Column wise combine of tab, comma, call or something.

Exactly. Right. Where that's a tabular column instead of a full tabular there.

Oh, you mean that is going to be implied that in that case it's going to join and row IDs without any specific. Like specific.

Yes. The only question is what's the name of the added column?

Yep.

Can we use a rule or do we. Did we already use up rules for millions of other things and column wise combined.

I think we did use rules.

I guess rules are used for prefixation. If we had. If we had just a bare column and use that in the association, that could be the name.

What about having to wrap the column in a tabular to be used to use column wise combine then you need.

To join it with like you need. You need some. Right now I think you have to. When you just give it two things. It tries to find common columns that it can do it right. You know, right. A join over and there's no common column.

I see.

But honestly, I don't see a problem. Maybe I'm missing something. To use the column instead of a tabular in column wise combine and use a rule. Because the only way we use rules are typically for prefixing. And since column you can parse columns separately, you can distinguish them from tabulars. So I don't see any clash here.

What does this mean here?

Those are the columns to join on.

I see. So which way would round would you put it? Would you put it.

Well, look down at the association syntax which comes a little bit below association.

But again with. With the association Then you have to name your tabular too.

Right, but see this. So if. If tab 2 were a bare column, think of pre. If you're prefixing nothing, then that's just the prefix itself. So that would be. The column name would just be beta.

But now you have to prefix alpha. I mean, frankly I don't.

That's the problem.

Right, right. But I forget if we talked about this before, but is there a reason why we don't just have a different function for this? It's like some sort of column append or something.

I think that would be much more intuitive for people to have a different column insert or a column append.

Okay, so let's just. Let's just. Okay.

And I think we had a discussion in that core transform functions. I think we had some things on that, if I remember correctly.

Now are we talking about. Just now, right now are we talking about the problem of just adding.

A.

Tabular column in general or the specific problem of doing a cast?

Because no, we're not talking about costs, we're talking about. We're just talking about add columns. Yeah.

Okay.

Like if I take a column out by indexing, how would I put it back in would be one version.

We listed the function in the cleaning section. If you open the cleaning section below.

That'S where it comes.

I think we're cleaning.

I think one reason why we didn't consider column append as a separate function at the end.

So is this not insert columns?

Yeah, I think it is insert column.

Oh, it is.

There's also an append column, but append columns.

And I think these functions could be great for matrices too because this operation appears sometimes.

I agree.

Now is the name there? If it's an exit. So for insert columns. If then if new is an existing name, does that replace that column then or what happens?

I'd imagine.

Yes.

Like, like imagine if you. If you want to do this casting thing, then you'd want to replace.

Exactly.

Yeah. Ok, so then replace columns and insert columns are the same if the.

Ah, there's also a. Replace columns.

Okay, there is one question I wanted to ask here and this is something we discussed before. Should we then if we introduce these functions, should we constrain the length to always be the same as the length of other columns? Because in. If it's not the same, then it's kind of a realm of columns combined more than.

No, right. Of course they have to be the same. Yeah, yeah.

So it's an error if they're not.

Yeah. Okay.

Right now the insert Column. There's no way. I see at least in what's there on to control where we're inserting.

I was thinking about that.

So we talked about having anchor columns and transform column. I think it was maybe. Or this could be an append columns and they just always go at the end. Oh, we have an append columns. So.

What was our mechanism for anchor columns? How do we set that up? That's in transform columns, right?

I believe so. And Right. We just had the column name. Not as a rule. Wait, so we. We had a list. Or maybe this was in column construct. Construct columns. Whatever the other one is. No, because that starts from scratch. Yeah, but I'm sure we had this.

Yeah, I'm sure we had it too, because we were talking about anchor columns. We're talking about putting it in a list. Yes, you're right. Putting in a naked list where it just says to give the ordering. Basically, you're. You're putting.

Right.

I. Right. I think it. It was a nested list and you put the anchor before after.

Right?

Yeah. No, what you do is this. What you do is. Is the following thing as you're reading through. Okay, so this is in. In the subject of anchors.

Okay.

Oh, right. It's. It's left to. Right now I remember. Yes.

Right.

So I mean, in other words, this is a Thing one is something that's actually in there, but you're not commenting on it and you're saying new one goes to blah, blah, blah, Thing two is the anchor. Right? Yep.

Right. And then you could have new two and new three, and they would both come after thing two.

Exactly. Yes.

Isn't this the syntax for tabular construct?

No.

Column construct.

No, because column construct.

You construct columns or construct.

Yeah.

It could be. It could be for either, really.

Right.

I mean, it's just transform columns is identical. Construct columns accepted. Just make sure to always include the original columns.

Right.

And modified.

Right.

I bet. Basically, in construct columns, this would just be a, you know, a shorthand for thing one goes to, you know, thing one ampersand or what pound thing one ampersand or whatever is.

You can do that? Yes, you could do that. Although it's construct columns will then have no other columns in it. Whereas.

Exactly.

This is more what we want with insert columns. Right. This will be. This will be something for. Transform takes existing columns. Still.

Can I ask something like, we never really had the need to do this with associations like a number of functions for inserting things in the middle.

Like you.

You can insert them at the end and Then shuffle them around. But usually you don't really care much about the ordering.

I think people are going to care more.

Yeah, I mean, Right, right. I mean, people certainly care about the orders of things in their spreadsheets and.

Yeah, they do, but what I'm saying.

They'Re gonna need to care about the order, so let' just move on.

Yeah, but what I'm saying is that these are conceptually to operation. One is inserting and the other one is reordering.

Okay, that's a separate thing. But, but, okay, we've got another issue. We've now got three functions. Okay. We've got all these functions. Replace columns, transform, construct, et cetera, et cetera, et cetera. They're, they're all doing the, the same thing, basically.

I, I, I, I don't, I don't, I disagree with that. I think, you know, transform and construct are sort of in one category. Now you can make the argument that insert, replace and append. You know, there's sort of, all three of them are sort of mixtures of those functions in the, in, in the normal Wolfram language case. And so maybe we only need one or two of them.

Okay, so wait a minute. What does replace do as compared to transform? I think they do the same thing.

I thought replace column was something that we were proposing to take tabular columns. Right. You replace one tabular column with a different one column with a tabular column.

Where we're not allowing transform columns to have a right hand side, which is a tabular column. Correct.

It wants to be a function always or you know, one of those shorthands.

For a, for a function. Even though nothing is arguably a thing very much like, I mean, so, so the possibility would be transform columns could take tab. And then, you know, foo arrow tabular column. I don't really object to that. Just given that we're allowing things like nothing, I don't really see why we shouldn't allow tabular column there.

Well, I think, Stephen, I think that conceptually we still think of whatever we feed into these rules at the right hand side as functions that are acting row wise and for nothing. It's just the shortcut for assuming that it supplied nothing to every row on that place on that column. But if you add the tableau column, then it's.

I get the idea, I get the point. I think that the nothing thing is a little bit, a little bit of a squiggle.

Yeah.

And I don't understand why the tabular column isn't something, I mean, what happens if you Index into it, by the way. Wait a second, wait a second. If you index into a tabular column with its row indices, what do you get?

What do you mean?

Single brackets or double brackets?

Single brackets. I bet you get the element right.

No, there's no single bracket thing, but the double brackets work. So part works as you would expect it to.

Why doesn't. I mean, but if the single bracket thing worked, then this could be just thought of as a function.

No, no, no, because it's normally a function of association, not the row I.

By the way, nothing is not a squiggle because nothing, open, close, whatever is still nothing. Like if you do open list and then nothing and you pass it in empty association, closed list, then that still works.

Oh, I see that. I didn't know.

That's nice. So that's not even an abuse of notation, but tableau problem kind of is.

Let's review this one more time. It seems. Okay, the question is, do we want a whole other thing called replace columns where the only thing that can occur on the right hand side is a tabular column? Right. In this case, the only way this could work is. Is. And will we allow this whole thing? Will we allow nothing in replace columns? Will we allow that?

I think it's kind of fine to allow nothing, but I wonder, this can be done with a constructor.

I'm sorry, what?

It can be done with the tabular constructor, like with the head tabular.

I mean, I think we could sort of avoid this if we just had, you know, an append columns or insert columns. Maybe append columns is better. Right. And if it's right, if it's an existing name, it replaces it. And you know, we could add.

What does it do in an association? Append an association or replace. Right, Correct.

Yeah, it'll stick it at the end if it's new and if it will replace in. In place if it's an existing. Oh, it did it at the end. Okay.

Well, that's what you would expect a pen to do.

But now the thing is, maybe, probably the lvalue thing is what you're saying Italy could be.

Or probably it's the association comprehension that does that.

Well, what if you put join instead of. Yeah, I didn't expect that.

I doubt join will work. No.

Association then. Okay.

I think it'll do the flat. Oh, right.

That seems weird.

Yeah, I didn't expect that either for joining.

Okay, let's ask this question. Okay, two questions. We've got a tabular, I've got a new ordering of Columns. How do I inject the new ordering of columns into a tabular?

Just ordering with construct columns. Which is also why the name Construct columns always confused me. Like with construct columns you can just reorder them.

Right.

So in other words, to reorder I would say construct columns.

Yeah. And the second argument is tab and then random sample of or of of your.

Because the right hand side by default is hash the first thing. Yeah, yeah, yeah.

But I, I mean, remember when we were starting to work on this, we also talked about having spans and all in there.

So we're not going to do that.

No, we're not.

We're not going to do it. It's complicated enough as it is. You can construct that list. By the time you've got a 3,000 column tabular, you're going to be doing things programmatically. How do you extract a list of the names of the columns from a tabular?

One keys column piece.

Yep.

Okay. All right. So the, the basically if I want to do this, I say column keys and then whatever spannery I might do is, you know, if I say this, if I say, you know, five to nine. Now how do I do this? I basically what I want is 5 to 9, comma, can I do this? How do I do this if I want to reorder these one to four, four, is that going to work? Well, there's some way to do this. No, that's, that's not going to work because it's going to be two levels.

Right, that's two levels. There's no, there's no simple way to do that.

Oh, but, but arguably with the fact that associations are kind of got to move on auto flattening.

Yeah.

I mean, I don't think it should work there, but I still think it could work in construct columns.

Okay, all right, fine, fine. That's a future enhancement. Okay, let's just agree what we're doing with this. Okay. Do we want, I don't think we need all of replace, insert and append.

I think just a pen would be, would be sufficient.

No, but why not insert so that you can use the anchoring mechanism, which I think is nice. I think people are going to want that.

Well, I mean, arguably append could also take the anchoring mechanism.

I think it's going to be obscure too.

That's obscure. And insert. It's less obscure.

Yes. I mean my one argument again to prefer append over insert is that. Yeah, right. In insert you normally need to give a position and yes, I agree here, there's A default at the end, which is what makes it feel more like an append than an insert.

But you could also replace a column.

That's true.

Insert could have a default.

That goes at the end.

Right.

I think insert is less confusing because. Well, the issue is do we need replace columns separately? I argue no.

Yeah, I agree with you on that.

Yeah, yeah.

I think, I think if we, if we make clear how this anchoring mechanism works, I think insert is fine.

All right.

One other question which I think mentioned earlier is what about l values?

I hate L values.

I mean, we have really nice L value stuff with associations.

Yeah.

And we also have a lot of stuff for like matrices.

Yeah.

The problem, Christopher, is that the. The part syntax for tabular has rows, comma columns.

Right.

We don't want to make l value for rows. We'd have to have the single bracket syntax refer to columns, which is kind of a big. It's a big change. It's deviation.

Well, no, no, no.

Right.

I mean, it couldn't be. This is the point that you can make. Tab part all comma, foo equals something rather, that can work. A tab of 12 equals something rather can't work.

I think if we make that syntax easy, it's a disaster because it's very expensive to replace a row, a single row. It's. It's cheap to replace a column. It's expensive to replace a row. I don't think we need lvalues.

Look, I think lvalues are going to have a bug trail and I don't think we need l values in the first instance.

Oh, well, just let me mention that in the future we might want to have them because.

I understand, but we're not going to have them at first, so we don't have to solve this.

Yeah, yeah.

Okay. All right. So we're agreeing on insert columns, transform columns, construct columns and insert columns, Right? Yep. Now, you know, what is all this stuff? We've got delete columns, which I think is a useful thing to have. Which of these are we having? Can we please be clear on this?

So I think there is a sentence explaining that some of them are actually subsumed.

Okay.

Yeah.

Well, now that we've had this discussion, I think we could make a much better summary, Steven.

Okay, well, please do. Please do. Did we have. We. Are we going to have this separate columns or not? Was that subsumed?

We decided that was going to be subsumed because we're going to allow the left hand side of a rule to be a list of a set of Columns.

Okay, all right, fine. Okay, can we please get a clean list and a clean guide page of all the functions that we're trying to have? Okay, we got to move on here. Okay, so we talked about element types. What was the big thing that we ran into last time that we were really stuck on?

It was the date parameterization.

Where, for what? Where is that now?

Date types. See, when we, when we do a date, we have, we have parameterization of how we're representing the UNIX time and then what the granularity is, and so on and so forth, which we need that information somehow to say how to convert back to a date object or a time object for the time case. And I think this is a nice way to do it.

But, but it is not in this section. Where was it? There was this five argument type or something?

Well, there is an example there in the tabular column that's shown. Right. So the, the idea is that the date type has the same freedoms of the days object.

So, so.

It'S a way of allowing users to, to specify the type. What are the freedoms? Okay, the parameters.

This is the type. It needs type syntax, which means it can't say date open bracket. It needs to say.

No, no, I know it needs to be type satisfyer, Steven, but that's. Yeah, that, that will be done.

Okay, so what's the question here? So then if, if date is a rather messy low level type that looks like this, I guess we can have that. Yeah.

And, and you know, it's, it's, it's a little bit different than, than, than what's been done in the compiler, but I think we can, we can make it work in the compiler as well with some effort.

Right. Okay, sorry. Okay, this. Okay. All right. Should we move on here?

Okay.

Okay. Really? We need to see clean guide pages and things of all this stuff.

Yep.

All right. What is this now? I did last time, we left over a bunch of stuff that wasn't completely finished, but I guess we now covered that. Been talking about the types and talking about these other operations. Is that true?

So in the previous meeting we were also discussing data connections, but I think that discussion will continue in another meeting.

Okay, so. Well, let's try and hit what we need to hit today. What happened to the visualization discussion? We needed that as well. What's happened to that?

I think, I think we need a discussion that's not in an open meeting. What do you mean by visualization? Do you mean the way tables look or do you mean plots?

No, I mean plots. Okay.

I think we need a discussion that's not in an open meeting first.

Okay, that doesn't sound good, but okay, fair enough. Then get that scheduled, please, as soon as possible.

So the final section is a set of functions we haven't discussed yet, which is how to do set operations. And it's something that we may not have to implement in the first, very first round, but it's good to look at. So the idea is to support union.

What is tabular data? What is that?

This is just a container. So we actually. If people start using random for every piece of data or programmatically generating data, then we're in deep doo doo.

So, right, this is the. This is the temporary example data for tabular. Yeah, we mentioned this last time.

Yeah.

Okay, mentioned this several times.

Could we not call it example tabular data just so we don't have to ask it 17 more times?

Sure.

So that was an example of a case in which just you and union. Right now we have to do normal, but eventually we will not need the normal.

Is that row wise? I'm confused by this. What is this doing?

Yes, this is taking unions and complements, et cetera, of rows.

Okay, Carlo, so wait a minute. Delete duplicates by means anytime the same species shows up on two different rows, squash those rows.

Yeah, you know, think. You know, I'm using normal. So it's really just what it would do on a list of associations. It's kind of unfortunate that in delete duplicates by you can't use a string. Like you can sort by. But.

Why can't use a string?

I don't know. I tried it and it didn't work well.

Okay, but maybe somebody didn't do the right thing in implementing it.

Yeah, I don't know.

What, what do you mean you can't use a string like.

You can't use like as bare key. Basically, in sort by, you can say sort by species without hash species and percent, just quote species.

Can you? I thought that only worked inside of data set or something like that.

Or maybe I'm just remembering. Maybe I'm confused.

Okay.

I always thought it should work though.

How important?

Definitely works in data set.

Let me. Let me just be clear. In version one here we've got a lot to do. Do we really need stuff like this there. This seems to me to introduce all kinds of issues.

It's not fundamental.

No, it's not fundamental because you can do that with an operation. You can basically do a delete duplicates by to get the indexes of the row and then use that part. So, so it's roundabout, but.

But I'm sorry, I don't understand that. I don't understand that. The whole point is this is a way. We've been talking about a bunch of column operations.

Yeah.

Okay, now we've got row operations that are doing things. I mean, how many other row. What, what are these other things here?

Control, Set operations, Steven.

On what? On what? On rows.

These are reasonable of associations in the end. So if you think of them as sets of associations, that's what you get.

Okay, but they're basically row operations. Correct? You're intersecting, you're looking at all the different rows. Am I correct or am I not correct?

Yes, yes, yes.

So let me understand this some more. So.

So if you open union, there's the definition of the sets. But basically I, I'm taking all the Chevrolet cars and I'm taking all the, the ones that have the, the 20 largest displacements and then doing union intersection complement those sets.

Okay, hold on, hold on, hold on. Let's try to understand them. Okay, so there we're taking that. I don't understand what things you're going to need to do normal on and what you're not. And that may be super.

Yeah, there. There were some things that weren't working, so I had to curse it.

I don't know what's working, what's not working, what's a proposed design, what's just hackery? I don't know.

See, the only thing that you need to care about is about the last line where you say tablet of union. I mean that's.

Is the normal there or not? Or is the normal in the end?

No, it's just going to be union of Chevrolet and large displacement. But for now, to show you something, I had to put tabular and normal because it's not implemented yet.

Okay, so the union is going across rows, basically. And is it sorting the rows or what's it doing?

Yeah, it's sorting the rows. By lexicographic order of associations.

Yeah, I get it, I get it. So, so I'm trying to understand.

I mean, it's exactly the same as union of a list of associations.

Right?

So if the entire row matches in both cases, then it's in there, right?

Is that.

Yes. And you don't need the tabular on the outside, Right? That's another piece of hack. Yes.

Right.

Yeah.

So this is just saying what does union do? On.

On two tabulars?

Yes, on two tabulars. And the answer is it should do row wise unioning Is that correct?

Yeah.

Yes.

Okay, I agree.

Okay.

And okay, so union should do row wise unioning intersection. To do rows wise intersection.

Okay, I thought that would be fairly uncontroversial. Now, now the question is what about same test?

Okay, but hold on, hold on, hold on. Yeah, well, let's look at same test in a minute and then we're going to come back to the delete duplicates by.

Yeah. I assume compliment is also just the obvious thing.

Yeah, yes. What other set operations are there? There are some other things like symmetric difference and things like that. I mean all the set operations should just work, should they not? They should just work row wise, correct?

Yep.

Yes. Including symmetric difference and so on. Are there any other set operations?

Guys, do we have set difference?

Maybe.

No, Compliment.

Let's complement.

Yeah.

We do have some metric difference though.

Yeah. Okay, so there are operations.

I think.

I mean, I don't think subsets or tuples is going to be particularly interesting for tabular.

No, no, no. I mean things like this. But doesn't have to be in V. One probably do want to be to work intersecting Q by the way.

Of course. Also all these could work on tabular columns.

Indeed. Yep. Okay. I don't know which of the. I mean these eventually need to work, but that's a different story. Okay, same test. I mean, I think a binary function in the case of union. Right.

Well, it's. Yeah.

Yes.

So why not support it here if we can?

No, no, I'm saying we should support it. But the thing is, what I. What I wanted to draw attention on is the fact that for set operations on entity classes, we've introduced same set same test properties, which has a slightly nicer.

We're not going to do that for version one. We're not going to get it right. We're not going to do it for version one.

There is another question here, which is if the tabular contains the key columns, should the default same test be still exact row match or should it be key columns match?

What?

Some tabulars will have sort of key columns which determine the uniqueness.

I don't. I mean, who cares? Look, the same test should just do the most obvious thing in this version we can view. Visit this later. Right now either implement the default same test or if you want implement the same test that does the obvious thing. Don't get fancy.

Yeah, I was asking about the default. Stephen. What?

What does it mean for two rows to be equal? So is it that they're same queue?

That the associations associated with them are same Queue in all cases.

Yeah.

Yes.

Okay.

Yeah. No, no. Weird thing about certain columns matter and other ones don't forget it. That's not what this is doing.

Okay.

Okay, let's look at Delete duplicates by. And delete duplicates. Okay, So I don't understand. I mean, delete duplicates, I guess you can have rows that are. I mean, one argument, union should work and delete duplicates. Might as well work and delete duplicate rows. Correct.

Yeah.

I think.

Who knows why there's a duplicate row that showed up there, but if there is a duplicate row, we might as well delete it.

Right?

That's a data cleaning operation. Do that.

Yes, yes. Now delete duplicates by. I mean, we could get more sophisticated with delete duplicates when they're missings and things like that. I mean, a version of that that would be nice to have is something that somehow takes rows. An operation I would like to see. It takes rows that have been added that are sort of the same row, but four missings, if you understand what I'm saying.

So you think of missing as an Annie pattern, like blank.

No, I think not, really.

I mean, I'm just saying that you've got two things that have, you know, A, C and F or something are filled in and another one where B, D and something else are filled in and where the C, you know, maybe B, C and so on are filled in where those agree. Do you see what I'm saying?

Yeah, I see what you're saying.

I was just.

Yeah.

I don't think Roger was disagreeing with you. Right. He was just saying you're testing by, you know. Right. You're basically doing match queue, where missing turns into blank.

So delete duplicates is uncontroversial. Delete duplicates by. Seems a little more controversial to me because that's saying. I see. Remove rows where, for example, they are identical by this column, for instance. Yeah. I mean, it seems like a useful operation.

Yes. By the way, the thing you were mentioning about the missings, there is one thing that we were talking about with Rob the other day, which is kind of similar and it's like. It's so. In SQL it's called coalesce and it's basically select first of non missing. So you feed it a list.

Yeah. Okay. That's a lovely operation. It's not for version one, but put it in the list for version two. Okay. All right. We've got to cauterize this design. Right. We gotta finish what we can finish. Go ahead.

I was just Saying, I mean, I would think more interesting then delete duplicates by would be some sort of. Right. You know, harmonize these rows. I, I don't know what we, I would call it or exactly how it would work, but rather than just dropping all except the first.

Right.

Rather than dropping all the entries that, you know are for Itaisev, it would be nice if, you know, I could reconcile the different rows.

Right. And in the end that's going to need to be done in ways where you're normalizing names and all kinds of things like this. There's all kinds of interesting data cleaning operations to be done there. I don't think we're quite ready for it yet. Okay, all right, let's review here. You know, we're trying. Oh, we got also the data connectors which we need to look at for a minute. But, but other than that and other than visualization, if I'm understanding it, we are closing in on, on finishing everything that we've talked about here. Is that correct? Is there something that hasn't been addressed?

Yeah, I, yeah, I think that touches a little bit of everything.

Okay, all right. Okay.

So we haven't had an in depth con. We haven't had an in depth conversation of, of typesetting either. I don't think.

No, we need to do that. Okay, so, so one thing that we get, we're going to need by the next one of these meetings is a nice clean set of guide pages where we can review everything we've agreed and where anything that is still exposed is brought up, which includes typesetting. What is the state of typesetting? Do we think we have something reasonable or not?

Yeah, I think we're converging on something reasonable. So I think that the, the primary thing that we've been focusing on is the sort of basics of typesetting right now. So like how do we display numbers so you can read them reasonably? How do we display dates and times?

Yeah, okay, but obviously Data Set had already pioneered a lot of these things. So I mean, there's also the question.

Yeah, I think we've improved on that quite a bit.

Okay, well, fine, we'll want to feed back that stuff to Data Set. But, but now just so I understand what we, what we probably close columns and things like this.

Sorry, what?

We've got to be able to fold up columns. If you've got 300 columns or something, you've got to be able to fold things up. That's something we had looked. I think Data Set has that kind of typesetting does it not now.

What.

Do you mean by folding?

Yeah, what do you mean by.

I mean that you can. That you can close. Can't you do that with dot, dot, dot and data set that you.

You mean hide?

Hide, hide, hide. Yes, sorry, hide. I should have said.

Yes, that's doable.

In.

In data set it. It's not yet in.

In.

In tabular and it's a double click that does the close. Does the hide.

I think it's the right mouse.

If it uses its right pore. It gets to close it or to hide it. Never mind. I'm sorry. It's.

We want to get to sort of. There's a whole new level we want to get to, but it can't make it for this version, which is to have kind of fancier tables. So that. Which allows one to communicate a lot of things like highlighting a column, a row, an entry, right foreground background extra.

Okay, but wait a second. Roger, Roger. I. We need to preemptively agree that the way that. That works in grid, which is highly functional but not easy to use, needs to be properly dovetailed with this.

Yeah, we. We're looking at that. We, we. We think it's not easy to use. 2.

It's not easy to use. I. If you want examples of it, go to endless blogs I've written that have tables in them that have highlighting and so on. Sure, okay. You can see it's. It works. But there, you know, it works and there are many issues about dividers and divider formatting and so on and so on and so on.

Yeah, I think it's sort of a little bit the same issue as this functions versus graphics. It's. It's a much higher level parametrization for this functions.

I know, but there's the look. I mean, again, there's a lot of examples that. And I, I can tell you what some of the issues are with. I mean the fact is there's a lot of stuff that was done positionally and so on, which probably should be done. I mean. Okay, let me give you an example. In cellular Automaton, there is a positional way of specifying rules. There is also an association named way of specifying rules that we added. I would have expected myself to use the named ones all the time. I do not. I, you know, can I explain that? In other words, there are functions like partition, for example, where we clearly need some named settings because it's just unrememberable how the positional stuff works. And. But I, I'm Just I'm not quite sure what it is. I mean, in the case of, you know, I don't know whether the issue is kind of a GUI is needed to design, you know, a table designer or whether it's. And the problem with that is that a, you know, a spreadsheet like GUI is not going to do us any good because we need something programmatic because table sizes change and things like this.

Yeah, I mean, we don't have a design of that to present. We have a lot of the use cases that we need to hit and there are several ways to do come up with a design for it. But it's not going to make it for now.

No, I understand. Not for this version. Okay, can we look at data connectors for a minute here? That was what we started looking at last time.

Yes.

Just a few minutes here to look at this. Okay, so give me a sense here. Wait a minute, wait a minute. What's this?

This is a separate section.

Which we don't have to talk about today. And we're not going to make it for this version, right?

Probably not.

I mean we. No, we can.

There's no way. Yeah, there's no way. We don't have time. Okay, so let's take a look at these data connection things. Which one should I look at first?

I thought there was also another. Is this the right thing, Sean? Or is there a better set of.

This doesn't look right. This doesn't look right. This better be cleaner than this. Where is the clean version?

Those details were just. If you wanted to know some information about the data set, open that back up.

I don't give a damn. I want to see the design. Where's the design?

The reference?

Well, the design is the ref page.

Okay, thank you. Oh boy.

This. I don't expect this be the long term look of data connection because I think we need separate pages like we have for import formats for each connection type.

No, I agree, I agree.

Okay, so the stuff that's in. In line on the page shouldn't be in the future.

All right, so what we had discussed last time. Where are my notes from last time? Where are the notes from last time, please? About.

I presume in the tabular folder. Carla.

No, the notes are reported on the. On the RP notes. But it's not like a document if you want.

No, no, no, no, no. I was typing in things like where we type, where we discuss data connection objects and so on. Where is that?

It's in. In Dropbox. If you look for the agenda from Last. Last meeting. Those are. Those are your notes.

I'm sorry, where in the agenda from.

The last meeting exactly. On Dropbox.

I understand, but for the 0509 meeting, is that correct?

Exactly. Yeah.

That.

That's the document you were writing on. And those are the notes you. You got there.

Should I be opening this in. Okay. All right.

Unfortunately, there are bugs in today's prototype, so the examples don't work. Sorry about that.

Okay, where are. Where is what I wrote here? Data connection. Where is this? Please make sure that when there are things. Okay, here we go. This is where we started writing this, right?

Yes.

Can we please make sure in future that when we have notes that got made here that they somehow get carried over and if they're really scratchy, you can put some background color behind them in the new version.

Okay.

So we don't lose them, please.

Yeah.

Okay. So what we had discussed before is that we have just like we have database reference as well as relational database. Relational database is the live queryable thing, if. Am I correct? Whereas database reference is just the authenticated hook. Is that correct?

I wouldn't call it live. Database reference is the result. Sorry, the result. The relational database is the result of an operation. If the database changes after you've evaluated, it's not going to go and call it again and say, hey, did you change? It's not live, but it has.

Okay, I understand that, but the difference is database reference is the authenticated hook to that data. Is that correct?

Yeah, it's basically a glorified URL to.

Connect to that, but it's not authenticated. It contains the authentication information. Great.

Okay. Is that correct? Okay, fine. And. But relational database, then. To get from a database reference to a relational database object requires that authentication happen. And then you are. You have an actual handle to something you can feed SQL queries to. Is that correct?

I don't think that's the best analog for this thing here.

Can you ask that question about databases? Is that how it works or is that not how.

Not really, Steven. The thing that relational database does is perform an inspection. So it's going to tell you whether and what there is inside the database.

You still end up needing to do an external evaluate to do a query.

No, you don't.

Yes.

You've got a million functions for doing queries on data.

Yeah, right. Of course you can do the query any way you want, but what I'm saying is that in theory you could do a query.

When does the authentication happen?

It depends, because in order to perform an inspection you have to do the authentication. But in theory, if somebody's told you that that table is present on the database, you don't need to perform the inspection to do the query. Do what I'm saying.

But then the authentication happens at the time of the first query or what?

Yes.

Okay, but then the authentication is cached. Correct.

To up to a certain point. Right?

It is cached, but right now it's kind of automatic, so it's cached on a different side of things, so to speak. Not in Wolfram language.

Okay.

It's always going to depend on the connection.

Right. Because. Right. Some connections.

Guys, guys. What we've got here is, is you have a data connection reference that has certain operations that you can do, like these listed queries. Is that correct? Is that correct?

So, data connection reference. No, I don't. Right now, all we have is data.

Okay.

You got to remember what we did last time. If I'm remembering what we did last time, this is your project. You should be able to remember, too.

No, I. I remember what we did last time. And you were talking about potentially splitting this up similar to kind of.

Not work presented right now.

So we already did some things similar that split up in external storage download and external storage object. And what we're finding is that having it combined makes a lot more sense.

Okay, well, I don't agree with you. What? What? You know, you gotta. You got to do better than just tell me that.

Sure.

So having them combined means that all of your data is in one place, which is.

Okay, what was the issue with external storage? You did not discuss external storage before.

Right.

If we pull up external storage download, an external storage object next to each other, there's nothing that external storage object kind of External storage download is just kind of a superfluous thing. That external storage object already kind of has all the information for.

Yeah, okay. That's programmer thinking. By the way, why on earth is there documentation here which has a missing argument? Can somebody answer that? Who's responsible for external storage object?

Probably nobody here.

All right, okay, fine. But that's a bug which somebody needs to deal with. Carla, can you make sure that's reported?

Yes, I will do it.

Okay, now let's go through this again. You think you can stuff the functionality into one function? I don't agree that that's the right thing to do.

Then you're going to end up with a different head for every kind of connection type. Right. This generalization means that based on the type, you can.

No, no, no, no, no, no, no.

What Are you up to none of us? Well, okay, I guess I can't speak for everybody, but clearly Sean and I don't understand what you're objecting to, so.

Okay. What I'm objecting to is this. Okay. Okay, let's go through what this needs to do. Right? We are not going to stuff things. It reminds me of you're filling a closet. Okay. Or a file cabinet, and you're like, yes, there's empty space there. Let's put something there. Right. Well, so that's what's the right way.

To design to be a list of like, strings or something. Like, is there a data connection and then some separate list of strings?

What are you talking about? Let's go through what this needs to do. Okay. What are the steps in data connection?

It needs to know?

There are, I think, two things that you need to be able to do. One, you need to have some kind of data connection, reference or object that represents the thing you can do operations against. Yeah.

You need to know where it is, what protocol you need in general authentication. And three, you need something that says what in that place that you need, like SQL. It might be a query string for a file system, it might be a path string.

This is such a hackalola, what we have here. No, this is not right. What the hell is this? Tables does this. If there's a table called tables, it fails. Come on, we can do better than that. Stephen.

We have an infinite number of objects in the system which have magic sub.

Values, but not like this. Okay, let's go through what it actually needs to do.

Stephen, can I make a. I also think that maybe there is room for improvement in the database stuff. But I think that if you think about it like you start from a data connection, which is data connection to a service like Dropbox, then once you've connected, that thing behaves like a file system. If you drill down to level of files, then those. It doesn't matter that they are on S3, on Dropbox or so. So what I'm.

But these aren't files. Query one is not a file.

I agree.

Well, yeah, but that this is a connection to a postgres database.

Right, but then this is something different.

Yeah, and I think. I think you have a point here. That the object at the database level is like the relational database. It looks only at metadata. So it tells you about tables, it can tell you about columns, but it cannot perform queries.

That's.

Drill down further.

Yes, that's right.

Yeah.

Which is why external storage object has things you can ask about the external storage object, but they are not the same thing as we would like download.

Round two to basically build exploration and other interfaces. But the. The ultimate thing that you could. You need in a report is something that gives you basically a link to data.

Yeah, I understand. I understand what you need. The whole thing can be built. I think this is not a difficult design. Okay. I think. I think we can do this. Right? The thing is, we can't stuff all these things together in one function. So.

So there could be things that generate this. Okay, but. But I think ultimately.

No, no, we're not talking about that. We're talking about one thing is there is the analog and thank you for the analogy, Sean, of external storage object. Then there is something that, you know, what we've got is we've got to be able to get the object.

And.

There'S metadata about the object, like its tables. I mean, nothing in this tells us. Okay, and then. What does this do on its own? Query one, without the import, what does that do? What does it return?

Data connection.

Right, right.

We went through strings and that returns a data connection. I don't think that's right. I mean, I think that. No, no, no. Oh, my God, that's so wrong. Okay, let's go through what this. Okay, one thing is the object with its metadata associated with it, like external storage object. How does one get. Okay, so for a database, relational database. If we want one table on a relational database, how do we get that? What's the representation of that? Somebody answer that, please.

You mean right now in the system?

Yes, in the system. What is a single table in the relational database? So I've got a single table. I've got the schema here. Yeah, relational database object. Now, how do I represent a single table?

We don't yet have that. That's what we plan to have once we get the tabular represented tables tabler delayed. Yeah.

Okay.

I mean, all that stuff that you're gonna see in that page is stuff about metadata on the table, not data.

Okay, how do I get the data?

Right now you can do it with the. You can do it with entities. You can do it with external evaluate.

You get the external evaluate way of doing it.

Yeah, we're talking about register an entity stored that. That calls that relational database, and then you can create.

Fine. So it's fairly roundabout. Okay, so in this case here, what we need is a thing which snarfs the thing in. So let me just understand what. What is the flow? Okay, first thing is we've got this authenticated connection. Second thing is, we are defining certain queries. See, what's wrong here is the mixture of this, essentially discovery interface with this. I mean, and again, what does this return? How is the data connection for this thing subscripted with query?

We went through this last time, Stephen. Right. It's another data connection where it, you know, has which one is the active query. Right. You can evaluate that and you'll see. I think.

That much should work. That much should work. The import will fail.

How do I get the. Can I say input form?

So there's query, goes to query 1. Saying which query? This in particular points.

Yeah, that's an absolutely terrible design. That's an absolutely abominable design. Sorry. But it's okay. We'll clean it up. Right?

What's so horrible about it, Stephen?

Because. Because you're mixing two levels of thing. You're mixing an active operation. You know. In other words, this is the thing that, you know, reveals metadata. Okay. This is something, you know, one way to think about it would be to think that this is resetting some kind of default.

Mm.

You know, the thing is, something has to actually have it take action.

Exactly. Which is why I want this to be as generalized as possible. So when you call import on this particular, you know, single connection, it makes a lot of sense. The alternative, I guess, would be clear. He goes to a single.

No, no, no, no, no, no, no. Because as you say, there are many different things. There are queries in databases. There are kind of, you know, bucket accesses or something in something which has data buckets. There are different kinds of operations. Right.

That ultimately point to some data.

Yeah, right. And. And the issue is, okay, so what we've got here, it reminds me, as we talked about last time, of the resource system. Remember that discussion? Yep, yep. Okay. So in the resource system, we already solved the problem of being able to say, let's look at resource objects. Okay?

That's something that we named and we created the metadata for and uploaded. So of course we can query that.

Steven, can I make a suggestion? I think that the main. I mean, the main problem I have when I see this input form is that it's like the father data connection. The parent data connection doesn't get eaten up properly. It gets all flattened. And I think, you know, it would be reasonable to have something which says path to a file, and then it has current data connection, Dropbox connection. I think that would look cleaner if you could keep the parent separate from the rest.

Yeah, no, I agree, I agree. Look, you gotta Have a thing which represents kind of.

So what does this look like? So if I have a data connection without that queries thing, what is the other thing? Aside from just like a list of strings containing the file paths or the queries?

No, no, look, look, the concept that there is a data connection which can be. Where you can discover that there is query one and query two and so on, that's fine. I don't think it should be called data connection. I think it should be called data connection reference or something like that. That reference may not be right. Okay, but this is, this is the thing that represents a set of things that can be done. Okay? Then the next thing and the things that can be done can have lots of discovery done on them. Like you can say what queries are here, what this is there, and so on.

So I mean, the thing, the thing I don't understand about, about this design, which nobody has mentioned thus far, is why are the queries in there? Right. Because I. Right. I should be able to create whatever queries I wanted in SQL. I mean, right, I mean.

No, no, I understand. I think it's reasonable to have some default queries in there. I think it's reasonable to have some, you know, things that you supply as and possibly even those are the only things that are accessible here.

We can have different levels of metaness. Okay. Or referenceness. And you know, there are going to be families of things where we can probably build a similar type discovery query.

Look, look, come on, we, we, we're over time here.

But the final thing that, that you need to have consistently packaged an analysis is this is my data.

Yeah, I know, I know.

I rehydrate it and I can go from there.

Right, I agree, I agree. So we need a thing that we can put in that is the, you know, okay, the, the best kind of thing would be something like this. Imagine the supply chain. Basically for this information, one thing could be you get sort of from the generic source. This is the basic way you authenticate and do this. And this is the URL for the database and whatever else it is, which could include. Here are some queries you might want to try. First level of supply chain is that generic thing. Then there's, oh, today I looked at the, you know, the scarves and hats section of the database and this thing that I'm doing is the scarves and hats thing. And I want to rehydrate the scarves and hats access to this database. Does that make sense? Yeah, Another level of supply chain. Right. And I think it will be useful. And you know, I think the only. Look, the big problem here is just trying to stuff everything into one function, which is not going to work. So I think what's needed is just two functions.

Right, Right. No, I mean I think with that's the long term goal was to have more than one level. This is just kind of.

Okay, so let's just figure this out. It's not difficult. Right. All we need to do is separate out the. Let me think about this for a second. Let me think for a second.

Stephen, have you noticed that, you know, data database connection. Database, sorry. Relational database and database reference and external object down. External object and external download. They're all still experimental. You know, if this was so obvious, why haven't we de. Experimentalized all those other functions?

Well, we might have done that at some point. I don't think we thought to do it, but. Okay. I'm not. I'm not okay at a meta level. I'm. I'm not meaning to be critical. I'm just meaning to get through this without being told that we don't need to get through it. Okay. This is not bad for a zeroth level design, but it isn't right. And let's not try and, you know, defend it. It's not hard, but we need to get it to something. Okay. My intuition is it will work with two levels.

Yeah, I also thought we need two levels and that sort of less concrete level could support all kinds of discovery.

Yeah, that's right. There should be a level that. Is this first thing here. Slightly different name and probably a different name, I'm not certain.

Maybe data connector or something.

What if it contains multiple queries though?

That's okay. Don't worry about it. Don't worry about it. None of those queries will be done by this function. It is merely declaring here are some things you can do.

Exactly. So I think that second level is already like import. It's already too. Data connection.

Just look, look. You can't have this. That will not work. That will not work. Okay, okay. That won't work. Okay. The, the. The first level is merely. Yeah, I mean, okay, this, this will take more unscrambling and we'll wait for another meeting. So do you see what I mean?

What in terms of. If there's more than one thing that is. Is there just like a bunch of data connections that are individual that only point to one thing?

No, I mean, I, I'm afraid, John, you're not making sense to me either. No, you're not communicating Your point? Very well.

Right, so. So one thing that you could imagine, for instance, and you guys know all this, like it's connecting to a file system and we all are familiar with discovery exploration interfaces for file systems. It could be to a whole relational database.

Well, I, Yeah, I mean, I mean, I have to say I'm not wild about the way that file systems work with their state dependent, you know, current directory and things like this.

Right.

No, the point that Sean is making is this, that he wants something where you've got this object that represents a query, one query into the static connection. He wants to be able to pick up that object and then do operations on it like import and so on. And that's a perfectly reasonable thing. And the issue is if the original thing at the higher level in the supply chain came with six common queries, you might want to do. How do you specialize that for one particular query? Is that a correct characterization, Sean?

No, it's more so if you want to get. You have one database and there's two different tables in it, and you want one tabular to have the data from the first table and the second tabular to have the data from the second second table in the database.

No, you weren't listening to what I said, I think.

Yeah, I think that's exactly what Stephen described.

Yeah, I think so too.

I think that last piece though, of like combining it into one query though. I want to get two distinct tabulars out of a single database, right?

At the same time.

No, I have one database, it has two different tables in it and I want to get both of those tables, tabulars.

Yes, that's exactly.

And so they each need a separate query to represent which table I'm getting the data.

Yes, yes, quite so.

And that's why this contains an association to multiple of them and you select.

That's right. We understand this, we understand it. The problem is this, okay, needs to be a thing that represents a single pipe. It can't be the same function, right, that has this one function that declares, oh, there are all these queries you can do. There's all this stuff you can do, feeding RAW SQL, whatever else it is, right? The second thing is a blob that is simply the unhydrated version of that data. See what I'm saying? There is a single thing that when you hydrate it, it will turn into a single tabular.

But Stephen, that, that, I think that's what this. I mean, that blob is going to be eventually tabular delayed. I mean, this is like much lower.

What is tabular delay?

I think the out of court version of tabular, whatever we decide to call it. Like you're not going to run?

No, I don't think so.

Hang on, hang on. Can I, you can take out nine and take tables of it.

That's the point.

Like data, that data connection has already collapsed into a single query, so you can't take tables of out nine.

I know, which is why it shouldn't be the same function. Okay, that function has to be a function where all the discovery interface works.

On it, but then they all get an individual name rather than getting grouped by types and things that then we're going to have a whole bunch of different heads.

I don't understand how you get from the former to the latter, but let me ask a slightly different question. So are you imagining something where you might.

By the way, by the way, Sean, just for you to understand, this external object can represent many different kinds of things. It represents many types. And so similarly we'll have some kind of external data object or something, I don't know what it'll be called that represents that thing and there will be many versions of that thing.

Can I, can I make a level?

Let me, let me finish my question to Sean. Are you imagining something where you could do import of. I'm not going to attempt to pronounce that P variable there and then list of query one, comma, query two and it'll give you two tabulars. Right.

I'm thinking let's have this in a separate meeting. We've got another thing we got to go to. Let's have this in a separate meeting. Let's, let's start next time by talking about this. Let's make sure we have a nice clean summary if you guys kind of pre talked about it because I think several people here understand what I think. We're going to be able to solve this pretty nicely with two functions. One, that I'm just going to say what I think is going to happen. We're going to have two functions. One is something very similar to this that is a discovery, that is a handle to the external data thing that is capable of having discovery done on it. The second thing we're going to have is some kind of external data object thing derived from this, which is a straight shot where you can say to it, just import it or whatever else. Maybe it doesn't even need an import. Maybe it's an open bracket, closed bracket, I don't know, but a thing.

So we're just renaming out nine. Yeah.

Yes, more or less.

Yeah.

Okay.

Yeah, that makes sense. That makes sense.

Okay, so. Well, it probably wouldn't be accessed quite the same way, but that. That type.

Right. Yeah. I think by renaming he means we'll have a head that has.

Yeah, like data object or something.

Yes. Right. So have a think through. I think that's going to work. Okay.

Okay.

My intuition at least.

All right.

Thank you. Okay. All right. Onward. Okay. See you all soon. Okay.