Live CEOing Ep 804: Language Design in the Wolfram Language [Tabular]

Foreign. Okay. Hi everyone. We are continuing our discussion of Tabula, and I think we're talking about data connection objects. Okay, so last time what we had discussed is this two level concept of. And has that been. Do you have a version of that here, Sean, or anybody else?

So this is that it has been split up into two levels. And for 14.1 we would just expose the object level.

Okay, so let's look at these two levels. Okay.

So then you'll want. So there's the link to the ref page for data connection object. And we just threw these together yesterday.

Okay.

I think update your docs.

Yeah, I need to update. Okay, hold on. Does it seem to work with the two level thing? Does that seem.

Yeah, it makes a lot of sense actually.

Okay. All right, so let me try to understand the. The. So we mostly looking at database connections or are we looking at other kinds of connections as well? Just to like S3 buckets and things.

Databases in S3 are, you know, the most important.

Explain to me an S3 bucket. Is it basically a file or multiple files? What's in it?

It's a bunch of files.

It's like, is it a directory? Is it a directory hierarchy? Is it a hierarchical directory?

Or is it one level hierarchical, has some metadata. They call their files objects.

Okay, what do I look at here? Data connection object and data connection, if.

You want to see both levels. So this is, you know, kind of the future plan for the discovery interface of making a connection to something and then getting information about like what tables are there, what files are there, etc. And then data connection object is the reference to a specific resource.

I see, that's sensible. Okay, so let me understand this. So you said. You then said for this version, you want to expose which data connection object. Okay. Okay. So in this case you would bypass the data connection by just saying database reference and then giving. This isn't quite right. Something like this is. Okay, look in the data connection. Go ahead.

You'll see kind of in the typical forms for source include. So that might be the path to the resource on S3 for a database. Right now we're just doing a string query because that's how you actually get the data into a tabular. And then Carlo has, you know, future ideas.

Okay, so the concept is a data connection object takes essentially a data connection as first argument, which could be elided. Could be kind of a direct database reference.

Yep.

Or for right now, you can call it Dropbox or just in an association, give it S3 and it's important information like region and bucket.

Okay, so then the second argument is the specialization where it's expecting, essentially it's an interpreter, that first thing is like a function. Basically, the data connection is like a function. It expects to be given one argument, which is its specializer, so to speak. And then the return value of the function is a blob of data that a correct statement.

If you import a data connection object, you'll get a blob of data.

I see. Data connection object on its own is. Is not yet imported, but it represents a single blob of data. A single expression worth of data. Correct.

Yep.

Okay, so let's go through what I was saying again for a second. Okay, so the way you're thinking about this is that second argument is essentially a function argument, am I right? Where the function means. Because I'm worried about the naked random thing there that represents a different thing depending on what the first argument is. Do you see what I'm saying?

Yeah, I do. So right now we're kind of playing with this experimentally and making it very direct as to what it's referring to and then seeing how we can expand on that. That's why we didn't expose this as like an association of named things or anything, because we want to play with this and really get that second argument right.

Well, maybe the right thing to do in defensive design right now would be to call that thing to put in an association and call it command or something.

Okay.

So I'm saying, I mean, just because that way, if it turns out we need three of them, right? So then this would become.

Yeah, Stephen, I think this is an interesting point you're raising. So at the very least, even if we're doing just SQL strings, it's never a good idea to provide a SQL string with the parameters already baked in. You want to provide it usually as template with some slot to be filled in from arguments. So already for SQL strings, I would think that an association with two things in it would be better than just a bare string.

I see. So, I mean, but then. But why aren't you then. Okay, now I'm a little bit confused. You've told me that a data connection object represents a single unique blob of data.

Yeah.

Okay, do you want that to be the case, or do you want it to represent a parameterized collection of blobs of data parameterized by a small number of parameters?

No, I think it should be a singular blob of data. But I think of that blob of data could be like. Think about it. Like imagine you're calling the API. You know, maybe that API is parameterized by something. But the thing that you're going to cache, that you're going to treat as an argument for import is always singular, even though by changing a number from 2 to 3 will give you a slightly different blob of data.

But look, look, I mean, we could clearly do this, right? You could choose to make the command be a string template just in terms of how you write the. The code, but you're not exposing that 70 to be fed to the data connection object in any way. The data connection.

No, you're not.

Right. And, and, and I think this is the right thing because. Because look, the data connection will already have done the authentication most of the time, I assume. In other words, the fact you put an authentication in this data connection object is something. Usually you would not do that. Usually the authentication would be taken care of by the data connection.

That's a convenient override if for just this application.

No, no, I agree with. There's no problem in putting it here. I'm just saying usually it would come.

From the data connection.

Yes, right. Which means you're not redoing the authentication, which means that the data connection object itself is a lightweight thing, which means it's not a big. Do you agree?

Yes.

So it's not a big deal to write code in the data connection object that makes part of it a string template or whatever else it is.

No, this is very expensive.

Okay, so.

I didn't catch what. What is expensive?

Expansive. Expansive with an A. Not expensive.

Ah, okay, okay. No, because you were talking about how cheap it was and easy to do things. How.

Okay, I don't know whether we should call it command or something else for that second argument. I just think. I mean, I don't know. Does anybody else have a comment about. Go ahead.

It might be context dependent. If we name it something. So that might be query, that might be path.

Then let's name it those things. I mean, since you know what it is for right now and we may just decide that if it's a plain string, we'll just do the right thing and that we want. I mean, let's remind ourselves. Do we have anybody from the resource system crowd here? How did this work for Resource Objects? How does this work for Resource Objects? Could we see if able to join us?

I've looked at it a bit.

Which piece? I got it, I got it. I'll get him.

Okay, what I'm talking about there is. I don't believe There is any unprotected string in a resource object. That is, I believe that the resource object has an association as its first argument.

And an association could be the first argument for this too, if you wanted to kind of give those more specific details for that particular connection. Well, that makes sense for that to be an association.

Yeah. I mean, so the alternative approach to this would be Data connection object. What do we call it here? Like resource location. Or.

So first argument is the connection.

No, I'm saying don't do that. I'm saying do this.

Okay.

I. This might not be right, but this is the analogy, right? This. And then, you know, query goes to this, and so on and so on and so on. So it's all just like in resource object. It's all just that as sock. Right. So everything's at the same level. Let me comment on why I think this might be a good idea. Okay, for example, let's imagine you need a cache location. Where would you put that?

I guess I had kind of an internal association with those kinds of lower level details that the user might not need to see.

Okay, are we going to be able to find Bob here?

I haven't been able to get a response from him, but I'm trying other channels.

Try Avery as well. Sorry, Ash Brown, he's out today. Okay.

An association containing those things does make some sense too. I mean, it's very similar. It frees up the arguments. I could make that happen.

Okay, well, but let's just understand what we already learned from this, okay? Because there's another thing here. Okay. It is not obvious that we shouldn't start. Data connection type, arrow, database, relational database, comma, blah, blah, blah.

So we thought this one through. And type is usually inferred from the connection. And then that's something that you can query as a property. And that's documented in the ref page.

Fine. Yep. But I mean, that's okay to have it be inferred. That's fine. You don't have to give it explicitly. The point is that, that this gives us the possibility that that is graph database or some such other thing.

And if you look in Data connection object, it's called relational database.

Okay.

To free up database for those other types of databases. Exactly.

Hi, Bob's here.

Oh, hello. We are designing Data connection object, which is something very similar to resource object. And we want to learn what we can from the design of Resource object.

Okay.

Okay, so remind me why we ended up putting everything in that initial association. I believe that the earlier designs had multiple arguments that not correct?

No, I think it was an association from the beginning.

Okay.

For the, for the. Like, for the, you know, standardized resolved form, you mean?

No, the way you specify.

Or, or if you're building one by.

Hand, do you mean. Yes, right.

No, I don't think it. People use, you know, we have these definition notebooks that people use to construct these. I don't think people are building them by hand with the association. So. Okay, I don't think that's been a successful story.

If we had. In resource data, in the data repository, you have some ability to access real time data. Is that a true statement?

No, data. The data repository is static.

It's not. No, but we had a design.

We do. Yes, yes, we have a design, but it's not.

What is the design?

Well, there's a couple different. Yeah, the design from a user's point of view would be straightforward. There just be specific content elements that pull from a live source.

Let's walk through that, please.

Right, so you would say resource data of the name or the resource object. And then you would have some element like you know, live, you know, stock price or something. And it.

Okay, let's walk through that because it's very relevant to what's happening here. Okay, so when you've defined a resource object, okay, you say there are sub cases, sub elements of the resource object. How are those set up.

In the definition notebook? Each one has its own.

Like, what does it turn into? What does it turn into here?

So for the non live feed situation, you know, it's. It's turned into an expression that's typically, you know, serialized as WXF and stored in a file.

No, that's not what I'm asking. I'm asking if there are three things. If there's, you know, A, B and C are three named elements. Right. How do you present those three named elements in the resource object?

Oh, how does the user like discover the names?

Now how is it written? If I'm trying to construct something, okay, so I say resource object and I, I want to make something which has.

If you want to build it by hand. I see you would do like. So you would do. Say it depends if you're giving, if you're giving like pointers.

Let's just say I've got. No, let's first of all do it without pointers and then we'll do it with pointers.

I think you would just say content elements as a string goes to an association where the names are the names of the elements and the.

Okay, right.

Okay, so it's going to fail if you don't give the resource a name and stuff like that. But this is conceptually what.

Okay, and what do I do here? Resource name arrow.

No, just name. Not resource name, just name. And you would say resource type goes to data resource.

That. Right, that's. Yes, yes. Why is it taking so long?

What is content elements?

Well, let's look at how they're accessed. How do I get the a?

So if this is right, you would just say resource data of percent comma a.

Yes. Right.

I don't think very many people are building this way.

No, I know they're building it with. With. With, you know, gu Builders. Right, but that's not irrelevant. I had no idea that this was even possible. Okay, so that's why I'm bringing this up because you said before that you might want three different query strings. Sean.

Yes, and we opted to, I guess split that up into individual objects.

Well, individual data connection objects. But the data connection, the original data connection might support three or four different standardized query strings, for example.

I don't think that's a good idea.

Okay, okay, fine. Because, you know, the way I see.

It is that, I mean, in the future when we have like an out of core version of delayed, you could just feed a whole whatever is called tabular delayed to this and it makes it into something that you can import, that you can cache, that you can rehydrate and whatever. But I don't think people are gonna write these things very much. But it's very convenient that it matches like file or URL as something that you don't feed to import directly.

Hold on, hold on, hold on. Let's unpack this a bit. Okay, so do we expect. Okay, so let me. Let's imagine in that scenario we've got a URL. Okay. The URL is real time now.com something/fish populations, whatever it is. Okay. And that's a thing that is going to return some. Well, it's going to return some data. Making sense. Let's say it's a WL file for now. But we want to call that, you know, what we want to be able to do is to say tabular of this. And how do we specify at what point? I mean, I think that should work. Basically implicitly it's doing an import. Go ahead.

I guess I would have just said that that's an import would return a tabular.

Well, I understand that, but let's, let's take the more the other case here where this is a data connection object A blah, blah, blah, whatever that is. I don't see any reason why this should be, you know, that should just run some discovery interface, produce some nice visual appearance, and then be off to the races. For you to do this as a lazy set of operations, shouldn't it? Why resolve this? Why not? Why do we need a separate tabular delayed? Well, for.

First of all, it really depends on what the underlying thing is. Because it's not like a WL file is something where you can do lazy operations on.

No, it's not. It's not. In that case, the only way this can work if you ask for any operation other than. I mean, maybe there's some trivial operation where that would work. Probably not, but I actually see it.

As the other way around. Like you have something which is not a data connection object to point to an external data source of some sort. Then you put tabular around it and that's the way you construct. Instead of writing select star from blah blah blah, you write it with nice stuff like select sort by and so on. And then at the end you have constructed symbolically something that is equivalent to the data connection object. At that point, you can choose to import it, or you can choose to use the data connection object to make it like a named thing that you can.

I don't understand. I don't understand. Why aren't we just doing this? Why aren't we just saying select here? I don't know. How do we do. How do we select a.

Okay, if we have a data connection object to a JPEG tabular, that doesn't make a whole lot of sense. No import of a data connection object for things that should return, a tabular will.

No, no, just, just a second.

Let me say one thing still right now, tabular is a very concrete data structure which is strictly in memory. So that's one of the reasons why we are talking about separate head.

I know, but I don't think it's right. I don't think there's a reason to do it that way. I think, you know, with image, for example, we have the notion of image and dynamic image.

But it is the same. Is that the same head?

Well, well, it's not the same.

Yeah, that's what Carlos says.

Right? That's why. Why is this not. Okay, let me, let me just try the following thing. What about the notion of a dynamic tabular.

That tells you dynamic image been a big success?

No, and also, I don't think. I mean, maybe I'm wrong. I don't know the details. If Julio is here, then he can probably knows them better. But my understanding was that dynamic image, right, the display is dynamic, not the loading of the image.

Both.

Oh, both are okay.

It depends if you, if you point to a file or two, you can use it to lazy display of an in core image, but if you point to a file, it doesn't import the whole file.

Why does somebody write something like this as an example? Why do they do it? Why is it written that way? It's stupid, if I might say so. Why is it, why is it not written in one line? Why does it, why does it define a variable?

Well, I guess we're all very used to define variables.

Okay, then don't do it. Just get rid of it. Can somebody fix that, please?

Yeah, I'll check that. I mean, the only thing would be if they. Maybe they had a version where they showed that it's a static image and then the next example they show that it's just a file path.

Yeah, well, right, that would be. Yeah, that, that looks to me more like either copy paste Titus or you know, editing that went awry. Not an intentional example. Okay, there's my dynamic apple. Now if what I say here is something like image take percent comma, I don't know, 50, 50, is that sensible?

I don't know how many operations were implemented on top of dynamic image. I don't think many. But I cannot really comment for the full extent of this.

Well, that's obviously wrong, right? It should work. But that's what we're talking about with tabular making stuff like that work.

Well, in this case, particularly easy because you can check with example data, test image. It should be, should be the same. But yeah, I see a point.

It should be the same. It should just work. I mean, that's why dynamic image probably doesn't have a lot of traction. It's because nothing works. It. Okay, but in any case, I mean, dynamic tabular tells you why you might use a tabular that is. I mean, otherwise. But it's.

It's everything but dynamic. I mean, if anything, the, the tabular that is in memory is the dynamic one.

This one is like, okay, doesn't really matter.

It never does anything.

Right. But, but look, a tabular, in my opinion of a remote database should. Its display should be based on some basic discovery done on that remote database, its default display.

Wait a second, Stephen, there's a mismatch here. Like tabular is not going to be on a whole database. It's going to be based maybe on one table.

Fine. Fine.

Yes, of course. That's what we're planning to do. It's basically, it's the same thing. So all the operations that we've designed so far in the transform language are operations that can be done just on the metadata. So, you know, you're manipulating the names of the columns, the types of the columns. You never know. Maybe there's like zero rows.

Okay, okay, okay. Can we be concrete here? If I wanted to get a single table, I've got this data connection here, right? That's an authentication to a handle to some URL to a database. Is that right? Yeah.

Or a database reference can go there.

Yeah.

Okay, fine. Let's put a database reference here. Okay. Now, am I right in saying there is a query you can give here which gives me a table from this. From this database reference? Is that right?

Or maybe, maybe we could even support just the name of the table there. If it's like select all the columns of that.

I don't want to be ambiguous with what the string is.

Maybe it's not the bear string.

Let me say, Let me point out. Guys, let me point out. You are slipping back into needing an association here.

I do think we need an association there.

Okay, fine. So one possibility would be. One possibility. You could say database table, arrow, name of table.

Yeah, something like that. Yeah.

Database query. I mean, I'm just making this up.

You have either one of those, not both.

I know, I realize you have either one. I'm saying you make sense. An association where one or the other is there. If you give both of them, it'll say error.

Yeah, or you could, you could even. Technically, you could even put there something like entity class, and then we turn that into a database query. We know how to do that.

Fine. Yeah, all the better.

Whatever's in the association, that data connection object obviously points to some resource. I still think that import of that is how you get back what the data is, regardless of.

I agree, I agree. Okay, so, but let's look at this for a second, right? So we're representing the thing we want with this data connection object. You want to get the big blob back, you say import. I agree with that. But now let's say you don't want the big blob back. You merely want this remote reference to the data. And the data connection object is the remote reference to the data. So why not let you just say tabular of that remote reference?

Because, Steven, I think that tabular should give you a concrete thing. Remember, like, I think it was like one or Maybe one and a half months ago, I showed you a demo of tabular delayed, which was just like it wasn't showing up as a table. It was something that would just tell you, oh, now I have five columns. These are their names. It behaves differently. And then at some point you want to make it into concrete data. So I think tabular could be the thing that resolves that or the dataset, if you like.

Tabular delayed aside, I mean. Robert, Roger, do you have any comments on arguments for tabular?

Well, I think when I think in terms of the, the outer core stuff, I want to design it when we can actually have real prototypes running and stuff. So we know what kind of trouble we're getting into.

Yeah, I, I, I, my comment was going to be, it's not obvious to me what, whether this will work or not. So this is not something to. Okay, but, but, but okay, I don't.

Think really, it's a really simple question here.

But the other thing with the, in terms of swallowing these, I'm thinking a data connection object will connect to many more things than tabular. But for, for, you know, clearly our focus is going to be structured data. So it's going to be tabular like data initially. But, but over time I kind of.

Dynamic image could perfectly well have a data connection object. Yeah.

For instance.

Right. But my claim is if you feed something like tabular or dynamic image for that matter, a data connection object, it'll just sit there.

I think it should do something. I don't think it should sit there.

Because it could do some pre processing. I just don't think it should. And I agree with him that if you want the frigging data, do import. Yeah.

And in one of my notebooks I have the word dynamic tabular of data connection written down because the idea of is if that data is changing, I want my tabular to change in real time too. That's the eventual thing that I would like to happen.

Right, but so the typical metaphor, the typical idiom today would be tabular of import of data connection object. Yeah. I mean I think what you typically.

Would be just import, just returns a tabular already.

Yeah. I don't think you need tabular around it.

I see. Okay, fine. Still, I think that's not completely obvious to me. That's not obviously correct to me. Tabby, in other words, look, look, look, look, look. I think this could have, it maybe has a default. It should have some kind of target in its association because otherwise, let's say one day you want it to return A dynamic image, then you're toast. You used up your import. It's not a bad.

Then it's an import element.

Yeah, that's fine. That's. That's what it should be.

Can I make a simple command, please? So it doesn't.

No, it doesn't need to be in the association. Just do import a data connection object.

It should be in the association. I think it is, because, you know, at the time when you create this association, when you create the data connection.

I think it's a great idea to have it in there because then you can override it later on and import.

If you want to default.

Yep, I see why.

Okay. You can. Yeah, it packages up nicely.

I agree with Bob. It should be default import element.

I. I agree. And we have this somewhere else, too. This is a great idea.

Yep. Okay. Leonid. Right.

I just wanted to disambiguate a little bit. Like. I think the part of the confusion here is that when we are talking about tabular delayed as a separate hat, it's because right now the action of tabular as it stands like tabular as it exists right now, is that it always contains data in memory. So we have a precedent in the lazy computations design where we basically constructed an interface, not a concrete representation with a certain head called lazy list. So my point is this, on the.

Other hand, audio, if I might say. So, when you say a precedent, I think it will be an antecedent, because that stuff hasn't been deployed. Yeah, audio.

I meant design precedence.

Tune.

Okay, Stephen, I sent you an email if you could open up that note.

Yeah, I will, but I want to hear what Roger had to say about audio first.

Audio can be both in memory and outer core. In core and outer core.

Right. Which is the same thing. We're.

And there's no. No two audio objects. There's no delayed audio or anything like that.

Right. It's just audio of file gives represents an audio which is out of core.

Heuristics, so that if it's large enough, we decide to do that by default.

Yeah, I mean, that's perfectly fine. But the thing is, if we make this decision, then tabler stops being just on a memory and becomes essentially an interface.

No, open that in 14. Otherwise nothing will work.

Because 14. Oh, as opposed to 14.1.

That doesn't need anything from the new stuff.

I know, but I don't have my version.

14.3 is fine. 13.0 is fine.

Probably 12. I'm doing 14.1 here because I'm 14:1, not prototype.

That that's what I meant.

Why?

Because some builds are some stuff in databases.

I don't have a version 14 available right now.

Okay, 14.1 is fine. Not prototype.

Okay. I just don't, you know, we already crossed this bridge of having in memory, out of memory with audio, for example, and so far it's worked well.

The thing is, with audio, the thing that is like audio is not explicitly lazy. Like audio, like the boxes look the same, you can play a file that's on your disk.

Absolutely. As, as. I don't think, I don't think this should be explicitly lazy either. I think the tabular, Stephen, if a.

Tabular delayed is actually pointing to a terabyte sized table on a base there, it has to behave differently in terms of boxes. In terms of what?

Here's what I think we should do. Okay. If you've got a thing that's in memory and has a hundred thousand rows in it, you're not going to display a hundred thousand rows, you're going to display a window onto it, right? With some kind of other meta information. I think the exact same thing should happen when you've got a remote terabyte.

That's, you know, the thing is that if you try start doing that and then you scroll the pages, first of all, it could take up to five seconds even just to get a window because maybe the query is complicated.

You know what, we can deal with this in the ui.

And secondly, Stephen, like suppose you don't control the remote server. You start paging through it, you get rate limited. Or maybe the data is changing.

Really irrelevant. I mean, it's irrelevant.

I feel like a lot of that stuff you would probably handle, if nothing else, by having an option that just specifies should this tabular show, you know, have fancy boxes or not. And maybe you could have it so that tabulars that are created by importing them from external things by default have it set so that it's, you know, doesn't have fancy boxes.

But I mean.

Okay, well, one argument though, that maybe this is different. Okay, so one argument. Why tabular is a little different from audio. Is that an audio. It's, it's. You don't really care how it's represented at all, right? Like it's just, it's an abstract piece of audio as opposed to here, where the whole reason you're using tabular as opposed to a list of associations or something, some of it is because there's nicer machinery around it, but a lot of it is because that, it's a. It's a more efficient data structure. So it's sort of, you know, if nothing else, you might want some.

That's exactly what I started with, that we have a choice. Either we associate tableau just with the existing data structure or we essentially convert it to an interface.

I think maybe one other, one other question then. Right, so if it's an interface, then what do we, you know, just as a test of whether we understand what's going on, what do we call the data structure? Like if we, if you imagine we had some function that tells you what's backing this tabular object. Oh, this one's backed by SQLite. This one's best, but.

Right, so then that's, that's precisely the. Yeah, that's precisely the problem that if you then want to deal with specifically the concrete data structure, which is currently.

Which is called a tabular expression, let's say.

Yeah, you'll have to rename it somehow any tabular expression.

I mean, that's what we call it.

That's, that's a bad name because, you know, when I have my tabular object, that is, you know, an expression with head. Tabular, I expect. Tabular expression.

Anyway, what I was saying is also there's a number of things. Like with an external database, you could have like maybe a thousand people concurrently writing to it while you visualize it. And so that's why you usually want to do everything in one block at the end and not. Oh, but it's. And then you evaluate and that thing is gone because somebody was writing to it. So, you know, it's not like something that's in your computer's memory and you own it entirely.

Right. And if you want it in your computer's memory, you do an import. Yeah, but we still need a data.

Structure that does work in memory, though. Like we have to do that first in order to do all those things.

Well, and there are other backends, I should mention there are other data frame backends besides databases, such as, you know, like somebody might want to use Pandas and Python as a backend things on that. Or Nvidia is working on a, on a GPU data frame thing that might be a possible back end.

Right. All of which should just work with tabular operations.

Right? Yeah, I mean, one, one, one thing though would be, I think some. Right. Some tabular operations. It'll be a little complicated because, you know, for, for the, for the in memory thing. Right. We do have some kind of magic compilation stuff where it's taking apart, you know, the, your Your column transformation function and so on, and, and then that can be, you know, turned into SQL or something. But it might not always be able to convert it like that.

And if it can't do it says, sorry, we're out of luck. Or do you really want to convert petabytes of data to whatever. Yeah.

Also, like, I think so far we've been pretty careful in designing primitives that would be compilable. And so maybe the thing that's not compiled like you're trying to compute vessel J on a postgres database, it doesn't know how to do that. But the whole. The heads transform construction, reduce and so on, those will compile.

So it's.

Well, right. Although I'm always a little bit sensitive to these magic invisible compilation things. You have some justification for that?

A lot of these databases have like FFI interfaces, so maybe some of this stuff we can push down from our runtime to the database. At least for SQLite, it's possible.

Yeah, I already done some experiments on that.

So I.

You know, regardless, I think that we're going to want a much richer tabular processing interface than those backends can provide. So I think that we'll run into many such limitations.

Yeah. And so most of the time you'll be slicing it and then, then it'll get into memory. So one of the issues is this. Okay, let's imagine that I had a tabular. Yeah, this is.

So I think that you're right, Stephen. There's a subset of things that will probably run on many of them, but then just, you know, by the time you've done that, you're down to something small.

Guys, guys, here's an issue. Okay, let's say I have an indirectly referenced tabular. And you know, a tabular of a remote object. Okay.

Yeah.

And that's very big. It's terabytes. Okay. Then I slice it down so that I've just got two megabytes of data. Okay. I'm referencing two megabytes of data. Now, having done that, I want to make sure that that is now in memory because I don't want to be messing around going to the external thing for my two megabyte operations. Yeah. How do I get it into memory? So let's say that, let's say in the design. Please. What's that? Slash, slash, please. Yeah. Right. Okay, so let's say that I've got import. Okay. But no, does that make sense? That doesn't make sense. Yeah, because the thing you've got. Let's Remind ourselves the thing we've got. The thing we've got is a data connection object. That was the indirect reference. The tabular is wrapped around it. Then to the tabular, we now say, you know, column, another tabular, whatever, and it returns another tabular.

Right, Steven? We had a precedent for that in that design of lazy list, because you had their lazy lists of several backends, so to speak. And what we decided to do, it was never done because that project is on hold now, but was to basically have the method or the type goes to something in the constructor. So you would just say tabular of this comma. Const. Like back end goes to.

Okay. Okay.

So.

So what you're saying is it's. It's a tabular of a tabular.

Yes.

Yep, yep.

So this is. And then I don't know what back.

End or whatever goes to memory or something like that.

Yeah, well, that's. That's like. That's basically data location.

Yep.

Yeah.

Something of that kind. Okay, that's reasonable.

And not only this, but it does. This also works the other way. If you want to transfer the in memory one into. Obviously this is not always going to work, but certain transformations may work of.

This kind between the back ends. Hold on. That's the case. Okay. Right. Okay, here's a question. You've got a data set and it's visible on the screen, or an audio. Let's take an audio, for example. Let me just do. Me just do this as an example. This doesn't crash or anything. Okay. Hi there, computer. Let's record enough audio to. Oh, look, this is now flashing. That's cool. All right, there's my audio. Okay. Now what I see visually here is storing cloud, whatever. How do I do that programmatically? The things that if I just want to take this thing and I want to store it wherever I want to store it. How do I do that programmatically?

Export.

Yeah.

Is that the answer? It's a little clunky.

You can click that.

No, no. What about store notebook? How do I do that programmatically?

Well, I assume the real programmatic way to do that is to set a very high limit for the storage in notebook.

Oh, that's clunky. What if I say iconize.

Sorry. Store. Store. A notebook is something that's tied to that particular box. Right. What would it mean to programmatically store that?

Those files, for example. Okay, but what does it mean? I want to get this data locally. How do I get that data locally? So that I can say.

Information Information.

Of object is in that file that you had their data inside.

No, no, I want to get it so I can.

Sure. Can you try information of the audio object? I don't work on audio. But this is.

Or you could import.

Right.

The thing. The thing that you want is you want a function that takes this file backed audio file and returns an in memory audio.

Well, then call normal on that.

Normal.

Yeah.

Okay.

Normal does random things.

It's not a very normal function. Data index.

I wouldn't have never. I would have never guessed this.

Let's see if that's.

Yeah.

Is it documented? I think so.

I feel like there's something where. There. It could be documented one of a hundred different places.

Yeah.

It is not command dash normal. It is not documented here.

No. And normal certainly doesn't document what normal does.

No, it is not.

That's normal, though.

Maybe it's parallel. That is. You know, how many times do we have to tell people you cannot put functionality only in examples. So this needs to be fixed. Who can take that message?

Yeah, I'm. I'm doing it.

Okay, well, that's a good answer. Now, in the case of tabular, because we got the exact same issue with tabular. So your proposal is to do something like this. And by the way, if we do that for tabular, it should also work for audio. A similar, you know, in. In incantation should work for audio.

Wait, sorry, which incantation?

This incantation. Audio of audio.

Oh, we don't use string options.

Well, we shouldn't use string options. I'm not. I'm. That. That shouldn't be a string option.

How data location relates to persistent location.

It. It's exactly the same thing.

Okay, thanks. Thanks for bringing this up because I need to use this for the vector database object, and persistent location is an argument is not an option. So if you want to fix this design, I'm.

Hold on, hold on, hold on, hold on, hold on. Persistent location. That represents.

Okay, so, yeah, you want to look at persistent symbol. Persistent symbol takes an argument, which is a persistence location. But right now there is no other function that creates persistent symbols.

But what does audio use to specify where it's persisting?

Audio model generated asset location. My guess. I know you've been in endless design meetings about that.

Well, maybe we need a thing called asset location.

Why does persistence location as a symbol even exist?

Because it's a wrapper that says so. This is wrong. So this would be. Let's say we say asset location here.

But I'm not sure that location is the Right thing here. Because one thing that it's possible is that you start from something that is backed by, say, SQLite, and then you back it with parquet. They're both on disk. They're two different types.

Whatever. Okay. Asset. Asset.

It's really more like backend type, Stephen, for the tabler.

Something like that. That's like an asset backed security. Never mind. But is it a toxic one?

Is the real question.

Indeed. Okay. Persistence location. Yeah. The reason we did this is because we needed to have a bunch of.

Because the path, persistence path uses a.

List of these things, right? And it uses. It has to wrap them in the type and the base and all those kinds of things. It needs a bit more information. Oh, it's a notebook. But where is the note? That's not the right answer. But you know, it's local. Where is it? You know what I'm saying? So that's why this is done that way. But I agree, we need. I don't know, it's called asset backing, but. But that's a stupid name. But you know, something like that. Backing. And that makes sense for audio as well. What. What are you calling it? Inside Audio. Do you know?

Look at the audio help page.

It's.

We're gonna say generated asset location, I think. I mean, we changed that several times.

Because it's a generated. Well, generated asset location is only for when you're creating it.

Where. Where do you stick it?

Right. The actual audio is just audio of. Right. You can see right there, it's audio of, you know, of file of blah.

Blah, blah, or audio of URL of blah.

Resource path.

Yes.

Yeah, but that, I mean, that is audio of file. Audio file, of course, is not creating a new file. The point is, if you do an audio operation on that audio that needs to generate a new file. Exactly. That is why it's called generated asset location. Because in principle, any transformation function will generate a new audio.

So if you allow this generated asset location in its options.

Well, because it's not out itself, that is, generating the assets is like, I don't know, audio X, audio Y, audio Z.

The operations.

Yeah, yeah, I understand. Okay. All right.

So for instance, that audio capture probably has it. I'm pretty sure.

Yeah, absolutely. That was our prototype. No, it doesn't.

Huh.

So let's try audio trim or something. Audio audio. What's a good audio time stretch? Why not that? Oh, gosh. It probably doesn't have its complete options here. Well, that's not him.

Oh, no, my. My bad. I was Mr. Umbering the generator location business was created for video. I don't know if maybe should have backported to audio, but. Yeah.

Oh, I thought, I thought it was. Yeah, I thought it was going to be and I don't know why it wasn't.

Okay. Yeah, Video generator has it for example, and, and our friends.

Yeah, this has it.

Yeah.

Yeah.

Probably video captured into every.

Every video function has it almost. I'm just, I'm checking now.

Well, you don't even have an in memory version of video, so.

Exactly. Video. Video was born out of core, so this was immediately the need to do this.

Right. I expect we're going to need to backport it to audio, but that's a minor issue here.

But the values of that are a specific file, not sort of like persistent space of local.

That's correct.

No, is.

It can be a directory, right?

It's a directory. Yeah. Look at that. $ Variable Name. What is this?

Okay, great. That's very helpful.

Yeah, this is, this is very standard for the like stuff that is resolved runtime. The front end is doing also this.

Because it is resolved based on documentation. Again, come on, you got to say what it's set to, please.

Yeah, this is probably resolved based on your operating system. A bunch of other things.

Fix it, please.

Or maybe what it can be set to.

Yeah, yeah, exactly.

But the first CC also link is generated task location without $ and yeah, you see that it takes the usual cloud, object, local, object, directory.

My point though is like for tabular, you don't really want to tell it a directory or all you want to say is that you want to.

Look, there are two cases here, okay? The cases. Three cases. You've got a remote location that you are, you know, doing things which are stored in the, in that same remote location. It's a database. You did a query in the database. The actual database server is representing the result of your query. Then you might want to snarf that all into memory. And another possibility which we've not discussed with databases, but it's not so relevant with databases, is what we're doing with video, which is you clone that database. You clone that slice of the database.

Yeah, that, that's, that's something that, that Sean has. I mean it's done under the hood for caching, but it is something like we. That would be done with parquet correction.

Yep.

Okay. All right, fine. So that, so there are the three cases.

Maybe we want to make it more explicit because I think that was like some kind of automatic caching for Sean.

Yeah. Okay. There are three cases we are dealing with here. Case number one, whatever happens is happening in your session of the remote database server, correct?

Yeah.

Case number two, you clone your own mini database or database like object, but it's still out of core. Case three, the full snarf and it comes into memory.

There is also case four, Stephen, which is you take something from the memory and you move it to some database table.

Okay, fine, I'm confused, Stephen. The case two, you're putting it on disk, but it's not in the notebook, correct?

That's correct. It's on disk in a clone of some kind, in a mini. In a parvo parquet or something like that. Get what I'm saying?

Yeah.

Okay, so, so let's define. Okay, so let's just define what we want to do here. So the claim is that tabular would be able to have one of these remote things. And tabular will need probably a generated asset location, because if you do an operation on the tabular that needs to generate an asset. So generated asset location would be none. If it's just an in memory thing. Generated asset location, if it's going back to the same place, would also be none. Am I making sense?

I'm not sure that's the way I think about this. Because, you know, in general what you do, it's kind of like what we did with entities. What you do is like you build up something. Like, you make sure you have something that you can gobble up in memory because it's small enough. And then you say something like normal, or you say something like import, whatever.

And we understand. That's why I went through these four cases. Let's write down the four cases. Okay, we need all, all of those cases. Case one, everything is done. Everything is being constructed. It's constructed in the remote database, but it's not.

So what I'm trying to say is that not. It doesn't mean that everything is being constructed in the remote database. It's more like it's being constructed locally as a lazy thing. And then you, you say, hey, there's another case.

We are. We construct our commands. They're two different things. We construct our commands locally with the idea that they could be implemented, that they could be deployed remotely. I think that's a separate orthogonal issue. Okay, the next issue is that the things that are being done are done in the remote database. There is data that has been assembled for you on a remote server.

Yeah, but that generally you don't do that eagerly. You always build it up in one thing. And then you send it all at once. You don't do it like, oh, first I do a select. Oh, no, that was wrong. Let me go back. You see what I'm saying?

So basically, the case one and two here, Stephen, that you. You listed here are essentially kind of the same.

I'm not listening to. I haven't got to my. I've got my case one here. Everything is being constructed in the remote database. Case two, everything. There's a. There's a pseudo clone. There's a remote pseudo clone being created. We understand what we mean by that, right?

What do you mean by remote? Like on your own disk, not on the database?

Well, I don't know where it is. It could be anywhere.

But these are live out of core connections, right? These aren't. This is the encore case.

Okay. Yes. Live out of core. Live. I don't know how to say that as live as opposed to live out of core. It's live. It's alive out of core. That's a weird English problem. Okay. Case three, it is fully snarfed into memory. Now. We had another case.

Yes.

That was basically, if you have something in memory and you're sending it somewhere.

Why is that relevant to this list?

The thing I don't. It's like, to me, we're looking at the. At the nodes of the graph where we should be looking at the edges.

Okay.

Yeah.

We got to come to closure on the things that we're doing for this version with some view of what the future is. My feeling is the future is tabular takes a thing. Tabular represents a thing which will display with a nice tabular thing which might have all the data, might not have all the data. It will. Then we need some operations which could be this. This idempotent tabular thing that allow you to. To move around where the tabular is stored. We might need something better than that. We might need a general way to do that. A general way to move assets around that will work for audio and video and everything else. Makes sense. But fundamentally, this will be one way to do it.

Yeah, that makes sense then.

Okay, so now what we're designing right now is the thing that represents the data connection object that soon one will be able to do tabular of data connection object and have the operations done remotely. But right now, it will be a pure import of the data connection object to get it into. To make it local. And as a kind of a little bit of a. Not quite a hack, but a side door, we're allowing a query to be done on the remote database through the data connection object rather than through the tabular wrapper. Does that make sense, people?

Yeah, it's pretty accurate description.

Okay, all right, so fine. Because in the future we wouldn't do any query at all here. We just say gulp it, you know, reference the whole database, and then all the querying is done through the operations done on tabular. Am I right?

Exactly.

Okay, all right, fine. So this is essentially a pre query. Now, do we want the possibility of a data connection defining seven prequeries, or do we just want to say you're giving a particular pre query when you explicitly create the data connection object?

You can just construct your pre queries as data connection objects in this.

Okay, so there's no reason to have a bank of pre queries that you've predefined associated with the data data connection object, can it?

I don't think that we need that now. And if we find that we do, we can add that later.

Okay, all right, so I think what we're concluding then is that the data connection object, what we'll do then is we'll say data connection object, And then we will have a data connection here which could also be a database reference. We will not put that in the association. I'm not sure that's the right decision. And then we will have things which are fairly pedantic here. Okay, here's a possibility. We do this as two arguments. This is the first argument. It's either database reference or data connection or something else. And then we have a pedantic second argument and we have an el cheapo second argument that is just a string that could just be a query string.

Sounds great.

Anybody else objections? So in this case, this will be. In the case where this is known to be a database reference, the thing.

That you would use as an end user is this data connection object. The thing that we want probably people in a repository for are data connections. So there are different types of connections to different services and sources and so on.

But my guess, more like data connectors. We're eventually going to need a data connector which tells you how to produce a data connection.

Right. A data connection is specific to a particular, you know, AWS bucket something or other thing.

And the data connector tells you how to connect to an AWS bucket.

I see, yeah. Okay, so that's data connected.

So for, for aws, it's a, you know, C backend. But that's kind of irrelevant. All you care about as the user is connecting to a file system.

Right. Okay, so Then. Then let's just talk about the supply chain for a. For a person doing, let's say, data science. Okay. There's, you know, essentially, let's imagine that there's a new, you know, a new cloud storage box drop or something. Okay. Somebody has to write the box drop connector code, which becomes the data connector. It's probably a service connection. Data connector somehow calls a service connection that came from, essentially, in generalized terms, the box drop vendor.

Whether it comes from a REST API or some other backing implementation constructing HTTP requests or whatever, there's a data connector interface to how that gets made, that produces the connection, which would be analogous.

To our service connections. It might even be a service connection.

It'll be built on top of service connections, in fact. Right, okay, I'm reusing that stuff.

Well, okay. All right. So. So there will be a data connector repository of those things. So then a person in it can.

Even really be the packlet repository. And certain packlets are just marked as data connectors.

Yeah, I wouldn't do it that way. I think we should have a data connector repository because I think we'll need special documentation breach of those things, just like we have a service connection repository. And not only that, but you want.

A way to find them instead of hunting around GitHub or something.

Right. Okay, so we got a data connector. Now the next level is you get a data connection. And for example, let's say that I wanted to access our transaction database, somebody would give me, I assume, a data connection. Is that correct? Somebody in corporate analysis would hand me a thing that is a data connection. Is that right? Yeah.

Or a data connection object, if it points to a specific thing.

Well, let's go through that. Who would do what at that level? I mean, so in other words, let's imagine that I want, you know, generic access to some, you know, to a transaction database. Then they would give me data connection. Okay, and then. But what if I want. Then if I want a particular table, then they would give me a data connection object. Right. And if I want a particular view of that particular table, they would give me a tabular.

No, but I think those are interchange. Like, really?

Yeah, they're interchangeable because that query can do a lot of combining of tables and whatnot.

Well, I don't like that they're interchangeable because I think it should be the case that at the different levels of the supply chain, you're expecting to get different things.

But, Stephen, think about it like, if you have a list and then you say, just select all the odd Numbers in there, there's still a list. And it's kind of the same there. It doesn't matter whether.

Okay, all right, fine. Yeah, but I think we've got. Okay, so we've got a reasonable plan here for what? I mean, does this make sense to everybody, what we're saying here? So in other words, the typical thing, let's just walk through what would happen. So somebody says, I want to show you, I want to give you a tabular that contains I don't know what. You know, I'm not going to ask for this, but you know, transactions in Timbuktu. Okay. Somebody's going to say, I'm going to give you that. I'm going to give it to you as a tabular. Is that a realistic thing?

And then it comes one import of.

A data connection object at the top of your notebook.

Well, that's one possibility. Or they just hand me a tabular that is lazily importing from that. From that data connection object. Just hand me the tabular. They say, here's a notebook, it's got a tabular in it that says tabular of data connection object, blah, blah, blah. Now let's just talk about authentication for a second. Okay. What does it actually do to take that scenario as an example. Okay.

When you go to retrieve some actual data, the authentication happens.

So I will get an authentication dialog that comes from an authentication whatever, our authentication. The authentication by default. Yes. Okay, fine.

And we made those changes to authentication.

If you're crazy enough, you could also put username and password in the authentication option.

Yeah, yeah, yeah, we know that. We know.

Of course.

If you pull up the.

Authentication guide or not. Get page, function page.

Yeah, you just added these things, right? Yep.

So there's a string dialog that would pull up an authentication dialog for sure. The automatic behavior does the smart thing and tries to cache things and then system credential is a no brainer.

Right. Which is. Which is stored in a keychain or something. Right. Okay, fine. That's pretty nice. Okay, so. So in other words. All right, well, I think we have a plan here. I mean, we've got some stuff that is. So for right now, I don't know whether we're going to insist that people use the idiom of saying import of data connection object. I think it's fine to say that right now, I think. And then for the next version, hopefully we'll be able to have remotely operating tabulars directly on the.

Stephen, this thing is useful for a lot of things that Are not tabular shaped. Data connection object could be pointing to a file on your Dropbox remote and it's. It's an image. So in that case, you definitely want him.

Yeah, that's fine. That's absolutely fine. 100% okay. But now in that case, we're imagining that data connection object will still have that second argument whose form is a courtesy string, whose form will be interpreted differently depending on what the first argument is. Correct? Yes, but the full form will always be an association that will say what each of the pieces is. Okay, all right, let's just look at these cases here. I mean, that's fine. That's a database reference of a file. We understand that. That makes sense. It's all good. Is this realistic? Don't you have to qualify that like 700 layers more? Who's Dropbox? Which one that you.

It will invoke the, you know, the browser sign in.

But how does it know that it's a. I mean, why isn't that. That should surely say this one's a little special.

This is. So. So right now Dropbox is a service connection and that is what I'm using to prototype building data connections on top of service connection.

So let me make a suggestion here. Rather than allowing this naked thing to work, just wrap that in the service connection object. Just say data connection object of a service connection object that you have already.

That's what we would love to do, but then I can't implement authentication how we just described because it would rely on service connections authentication and I have to catch it beforehand and pass it along. So if I. If I already have service object, it doesn't work.

But why does this connection, if you put the service connection in there, doesn't it request authentication on first use and then cache? Yep.

We are working on upgrading service connections. It's just not going to be ready for 14.1. Okay, authentication is going to work the same here.

Let me clarify. Service connection will. Will do what you expect, Stephen. What it is not doing is the enriched authentication that we started using for LLM functions and Sean is using for this thing where you can use all sorts of credentials.

Okay, look, I think we should document the service connection case and you know, hopefully we can pack that, update it quickly and you know, we can secretly know that this case works if you want. I just don't think we should leave that out there in documentation because I think it's a hack.

Julio, is that possible?

To do what? Sorry?

To do packlet update of this to.

Get the new service framework out for Dropbox.

Well, I mean, the new the service framework is a packlet, so it can be packet updated whenever we want. We made it the packet already in the previous version.

Okay. The answer is to get it. For this reason, I recommend that we document it as a service connection and we try and get that packet out as quickly as possible. Okay, what's this? Use the cache. Right.

So this is if you want to. If the remote resource might be updated. Somebody changed the table on the remote database. Redownload will force it to get it from the database and not the cache.

I don't like this way of doing it, but let me think for a second. What precedents do we have? Okay, let me just check because we're going to run out of time here. Let me just check what else? Okay, this is the last thing we have to discuss here is the way that redownload for caching works. Okay. What about the local cache function?

Local cache is if you want to kind of cache some specific expression and whatnot. And so this isn't particular.

This is the opposite issue. Right. It caches by default and you want to force a reload. So let me make a suggestion. With local cache, I have sometimes wanted to do exactly the same thing. That is.

Yeah, I think that this should. I think we should add this to local cache.

Right. But. But why not have a function just like we can. Why don't we just have redownload as a function or recache as a function? I feel like local cache does that. Right. I think if you do, it's based on the timestamp. Well, my understanding is that if you use the local object that's generated by local cache, then you get the cache version. But if you call local cache again, then it generates the cache.

Oh, okay.

Reevaluating local cache fetches a new copy if necessary. Only if necessary. Or if necessary. Oh, only if necessary. Or if no modification time is available. So it does. It does a ping to see the modification time if necessary.

Yeah, that's what I thought. So there is no way to actually control this?

I don't think I agree. It's not giving you a separate independent way of reevaluating. So how would you feel about a function, or maybe it's crazy, called recache, that's applied to a data connection object.

No, because you want to potentially make that a part of the object.

Right, I see. You want to do it every time.

Yep.

So effectively it's a. It's almost a dynamic data connection object.

But only for this object. If you set it and that gives you control.

Isn't this like no cache goes to true or cache goes to false? Something like that.

Sort of. But we want to trigger an action of actually getting the data in this kind of lazy data model interface.

I understand. It might also be true. Okay. You know, it's a little bit more subtle because. Say that this references one table. Okay. And your outer tabular, I mean, sorry, it references a whole database. Your outer tabular has specialized to one table. But you can easily tell by a ping to the database that that particular table has not been updated. So there's no reason to redownload that. That table. Do you see what I'm saying? Like the modification date of a website.

Well, we're looking at the local cache, so we're only going to. We're always going to pull the data when redownload is true. We're not doing any ping to see if the data's.

No, I know, but I'm saying you might want to do a ping. In other words, just like in the case of local cache, it's doing a ping to see if.

Let me think. It seems sensible. I just need to sit on it.

Okay, all right. But the point of that is. It's like. Okay, look, look, for right now, why don't we do this in the association that we're setting up up here that has all this information here. Have, you know, redownload. I don't know what all the different cases are. Goes to true. Just put it in that association in the data connection object.

Yeah, we can start there. I think that. I was thinking that set options might be useful on this or something. But let's start with this and.

Fine, yeah. What's the right name for it?

Refresh cache? Maybe.

But it's. I mean, is it use cache false.

But that doesn't indicate that we are triggering an actual acquisition of the data like redaction.

First question is, are you going to have a cache at all? All because you might have no cache. There's no point in. If you're going to redownload every single time. There's no point in caching. Right.

Well, we do still technically cache because we need something to point to to import. We have to retrieve the resource and then import the resource. So there's still a temp file anyway. And the point of this is to abstract away temp files.

I don't know. That's not a great name.

No, that dynamic seems to indicate that. Check the ping if the data is available. Type thing.

Right.

Well, shadi and I went over a few names on this and again that doesn't indicate that for like import of this I want to redownload because it can be specified at import time or it can be a part of the object.

What if we make the option not boolean like always? Never if necessary.

Yeah, I mean this is a hacky thing right now because we don't really understand it, but yes, that's a possibility. Oops. Or all automatic something like that. I don't know. We're close here. Something like this.

Yeah.

Okay, very quickly, what are these other things here?

Okay. Replace missing is one. It's part of a data cleaning suit is basically kind of an updated version of synthesized missing values which is a bit saner and works with tabular in mind like this, with column based data in general in mind.

Well, I don't like the right hand side is a function here because replace usually like delete missing things like that. People are going to think it's just the same as delete missing.

Delete missing doesn't take a rule.

No, I know it doesn't. But I mean this looks like a replace. Whereas that right hand side, what we've had before, I would say that a better thing would be just like it looks like we have transform columns will be transform missing.

So what does this actually do?

Right. I don't understand what this is supposed to be doing because when I see replace missing in a rule, I presume that means if I say and I think this is, I replace it with the right hand side. Yeah, I think it's just filling.

It's another replace is another name for filling.

Yeah. We floated fill missing as an option but we wanted the sibling function for anomalies or outliers.

True.

So I would, I would think about this in tandem with the function that does.

That's a good idea. That's a good idea.

It does what? What is it going to do to the anomalies?

So outliers and anomalies, you might replace them with like a clipped version. So like compute.

Okay, I understand. So but why not say transform missing and transform anomalies?

I might warm up to this. Let's write down some code with transform. I think it's not too terrible as.

A name in the future. Are we going to have a version of this that looks at the other columns and does something fancier than just like min. Like are we going to have something that fits the model and then impute.

Yeah, exactly.

This is the point that we are presenting here. So this is a Design point. So for the simple case, and not actually, not just a simple case, but in many very common cases, it would be very convenient that what you pass to the function is the actual column you're operating on. Now this of course makes writing a lot of stuff easier, and that's fine. This is a specialized version of transform columns or like a tabular operating function, let's say that is done for a specific purpose. But the problem with this is that you make it impossible to do a fancier operation using the full tabular. So our proposal for version one of these is to just allow special syntax like strings and literals. If you open Stephen the documentation page, there's a bunch of stuff that this function can do.

Am I correct in saying transform columns takes a column name and a function, Right? That's how it works. Yes, yes, yeah, yeah.

New column and the right hand side.

That function always works on the row element wise by the rows of the tabular. And these functions, you know, like to get the mean has to know about the whole column at once. So it's a different kind of semantics.

But the other thing allows you to reference the column with a hash, whatever it is. Yep. Because it's as if the association.

Yes, the hash, whatever in transform columns is the current cell.

I know this will have the same.

Syntax as reduced tabula. It's going to be column wise.

And is the plan that this will replace synthesized missing values or we will have.

I mean, it depends on how it evolves. The main problem here is that this needs to work column wise because it needs to be able to do operations on the column. But. And just to prevent what Karl will say in a moment, you want to know for each row, ideally where you are. Let's save it. A very common operation that I can do with a string setting is taken.

From the one above. Exactly.

Or taking from the previous one, the first non missing or the next non missing. So that is very easy to do if we have windowing, but we don't have windowing yet for tabular operations. So I think this. I like this suggestion that came up today of only allow string and literals. I mean literals can be. We can support innovative. Say column goes to zero.

Exactly.

It's easy to disambiguate. And then I have that syntax which is constant comma value, which is a quoting mechanism for any type of literal that you want to put. So if you want to say constant comma interpolation, because you want to literally place interpolation there, you can do that. So this you can do. And I would wait for the function syntax, because the function syntax will definitely benefit from some advanced column manipulation in tabular that has a concept of where you are. Like the current index of a sort of windowing. You might want to look at the previous not missing in 10 elements. So that thing is something that we need to develop yet for tableau in general.

Yeah.

One thing you might want to do is say, look at the example you have there. The previous is not missing of the same color as the one you're in right now. So you have to first group by colors.

Yeah, that's hugely complicated. You know, logic in general.

Yeah, exactly. But this is not just for missing.

This is.

This is true for any tabular operation that we do. So this. We need to decide this syntax. But it's not something easy to decide in a day.

Right now I don't need it just for tabular. Like, I would really like if one of these days you want to invite me to an ILD meeting, I can explain what windowing does and what. Where we can use it.

No, I agree. And it's. It's. It's similar to the. To supplied. Supplied is. Is going in a different direction. Is going out into. In to make a stupid physicsy thing. It's like the internal degrees of freedom are being accessed by supplied windowing is accessing the spatial degrees of freedom.

Well, you want both in a general case. Yeah.

For something like this. No, I agree.

It's actually a generalization of supplied like supplies is a sub case of this really. But again, I'll show you.

Yeah, okay, fine, we should do that. Okay. But for right now, here, as an experimental function transform missing, I have to say, you know, my preference would be to just say, oh, I don't know. I worry about using that rule syntax for a list of columns. Rule syntax.

An association.

Yeah, that's less suggestive. Well, but what would you put in it?

The same thing. Exactly the same thing.

What would you put the column. But you're doing this for multiple columns, right? Or are you doing it one at a time for each column at a time? Look, I mean, what you could do.

Then I have. If you scroll down, there's still something that would be nice to do. So there I'm saying when specify desired column call. You can use the following thing like a number, a name. Then this is an abuse of type. But I really think that we want. You know, we can do whatever, but I really think we want something like that.

You cannot get this design. This is a Complicated design. We're not going to get it done correctly right now.

Okay, I'm just throwing ideas here, but just to at least start thinking about directions.

You're definitely throwing ideas, but you're, you're certainly not arranging them yet.

A way to work around this for the short term is to have something like replace missing. What does it do on a transformation?

It's not going to be replaced, transform missing, whatever it is. Right.

Transform missing. Suppose we obvious. We probably want a version that works just on tabular column alone, right?

Yeah, I, I think, look, I think we could say the defensive design right now would be to just say columns. No, it'll be something like this, a arrow, you know, action arrow. Is that right? I don't know. I, I mean, try and basically festoon it in, in associations, I think, because if you've got that thing with types and so on, it's going to be a complicated mess to get all that stuff in there. If you make the whole thing in association, you just say, okay, here's another point. You can always nest transform missings, so there's no reason to have multiple ones happen at the same time. So you can say columns arrow, blah, action arrow, blah, blah, blah, blah, blah, blah, blah. And if it's the same action on multiple columns, you can just go ahead and do that. But if you're doing different actions on different columns, then fold a bunch of.

Transform missings or, or give a list of those associations.

Maybe, but I'm worried about using UP syntax. Yeah, of those. But, but I think that's dangerous. That's what I would do. I would, I would make the thing as explicit as possible.

Okay, so let me refresh this page for next time and I'll have a prototype implementation and then we can see if it's satisfying.

Is there an issue with this? Is this something we can do in a minute or is this something that's going to take.

You agree?

Slightly complicated, but what I'm asking. So I think your Christopher's idea of the association semantics for when it's given as association is a good idea. But I would like it to be consistent with the list case, I. E. The function only takes one argument in either case because otherwise it's completely inconsistent and people aren't going to know what to do.

Do you have a way to set that up? The, the one argument in both cases?

Yes, yes, it's easy to set that up and you can work around it by that very last thing I have is you just put an apply around the function which Converts from bear arc to 1arg. So if you agree, I'll just.

The issue is, could you remind me, what does column wise. Column wise combine? Takes in general one column and applies a function. What. What are these?

It mer. What it does is it. It merges a bunch of diff. A bunch of tabulars and. And often. Right. So. And the function tells you the most general case is the function which tells you when to.

Okay. The alternative form of this was that this list goes to a single function which is automatically applied to the elements from each of the. Each of the tabul. Is that right?

Right. Right.

Okay, well, I. Fine. Do the apply.

Okay.

But if you. If you have a chance to describe this to Christopher, who I think had to leave or bring this up next time.

Yeah, I'll implement over some examples and. Yeah. And then.

Seem reasonable.

Yep, sounds good to me.

Okay, so Tyrus is commenting. Why not name this? Impute missing. Don't we already have that? No, because the whole point is that we're. It's impute missing values and. Which is not the same thing here. We're not imputing. We. We have no idea what we're doing. We might be just slashing and burning the missing values, so to speak. Dan, earlier in the live stream asked, will data sets and data repository be updated to tabio when relevant? Absolutely. And that's something we all need to do. Yeah. Okay. We're making progress, guys. Great stuff. Thank you up. See you later.

Bye now.