Live CEOing Ep 817: Language Design in Wolfram Language [Tabular]

Hello, everyone. Welcome to another Wolfram language design review. We had a bit of a hiatus in these things as we. As I was involved in our summer school and high school summer research program for a few weeks. And so we were. I was able to be involved in less of this, but everybody else was doing all kinds of things, and I expect to see lots of cool stuff here today. All right, let's take a look. Should we start off with the status reports?

Yes. So these are brief things, brief notes that developers have sent.

Okay, so what is done?

So this is a list of things that are done. And that means we have to do a lot more testing, but it should be working.

Okay, so give me a sense. It's hard for me to scan this. Give me a sense.

So we have the basic objects, tabular, tabular row, tabular column working, and then a number of operations on them like sorting. And the last things that Rob was implementing was adding new backends. And Carlo is already making use of that and error handling.

Okay, so this is true out of core backends?

Yes.

And yeah, Carlo has some examples at the end of the agenda.

Okay, that's exciting. All right, now, Okay, so hold on. Are there decisions that we have to make about some of these things?

No, I wouldn't. I don't think this party needs discussion today, and particularly we need Rob for any discussion. There are a few details to discuss here, but we should wait until he comes back.

Okay. Okay. So what aspect of the error? What is that error handling? I mean, what's happening now and what's.

Yes, so Rob is using an error framework that Leonid and others have created and seems to be working well. And we just need to propagate that through all of the functionality. It's one of the last things he did before going on holidays.

Well, the thing that is interesting about this is, like, it's not something that per se, changes much in terms of user visible functionality of Tabular itself. But on the other hand, it. It's the result of work that has been done by several people, including Christopher, on the modernization of how things are handled in the Wolfram language. In terms of, like, you are a developer and you want to write a large warframe language project. How does handling works? And we're using Tabular as a testbed for that.

Really?

So let me ask you this. I mean, in terms of when there is an error, to what extent are we telling people, you know, user error failure, and to what extent are we giving them a pointer for how to recover?

Well, I think that's a very orthogonal question to whether or not we're using an error framework. It's more about the style in which we use the errors.

I understand.

I don't know how much of like actual error writing Rob has done, but I can say on my side, in the out of core stuff, I'm trying to as much as possible say things like, I see you're trying to compile plus between an integer and a string. That's not going to be possible on an external database. So that will.

Okay, but are we giving them. Because, I mean, generally the general style for the last three decades of our error handling has been along the lines of this is what happened. We can say no more. But I think we are evolving to a point where we are, you know, where we have more cases where we can kind of know where, you know, what direction you should go from what happened? Do you see what I'm saying?

Yeah. So in that last example that I just said, you would add something like, by the way, if you want to turn a string into an integer, use from digits maybe.

Yes, I think we should review a bunch of these error messages and just see.

Yes, the last thing we had discussed is allowing local control, like having functions from inside the code and missing fallback style and things like that, and then having global control, like having a dollar variable or an option in which you pass an association of parameters, saying, how do you want. How you want tabular to behave in the case of an error?

Yes, but I mean, I think, you know, this is something we should discuss in user assistance. But I kind of think that we should be starting to think about auto auto suggests around error messages. Like in the case. You just mentioned that it has a little, you know, I mean, I button or something.

Yeah, I mean, okay, one for the LLM for like chat notebooks. We already do a version of that. But honestly, I think that I would want to see a use case where you can actually provide something, a useful suggestion in the I menu that isn't just something that the primary developer would have already implemented. Right. Like if it just says, you know, oh, you gave me a integer when I was expecting a string. Like, well, then, you know, maybe that's a. Maybe that's a fixable thing. Or maybe the original developer should have just made it so that it's willing to stringify an integer if you pass it one. Like, a lot of the problems really have such an obvious solution that you can just tell the user to do it. The code might as well do it already.

I understand I understand. Your point is, if there are no off ramps that you could take and you're just, you know, you just have to go on those rails, you might as well just make the code do that. The question is, are there.

Yeah.

Other side. I don't know.

I don't know.

The train, other junctions, if there are junctions you might want to take, that might be worth asking the user, do you want me to do this? Just like, it's like, you know, you're quitting, you know, do you want to save a file or not? I don't know.

Well, I mean, asking programmatic code to, you know, ask questions.

Right.

That's obviously very problematic because.

Right.

Suppose you have some big workflow and, you know, you hit shift enter and then you walk away.

Exactly, exactly. So what I'm saying is, well, there's a UX approach to this, which is the message just says, I'm sorry, these types don't match. Okay, but there's an I button in that message that allows you, or some kind of autocomplete that allows you to do things where, I mean, just like you could say, you know, you're loading code from a file the auto suggests are not coming up as you load the code from the file. See what I'm saying? But if you happen to be there, you can have the possibility of, of, you know.

Well, let's consider like a real case. Right? I mean, the most common. Okay, so in general, I think that the argument has been to not do this sort of thing because if you tell the user, try doing X and it turns out that's that's wrong, then it's really frustrating. So take the example that you were giving about. If you give it a string, it's expecting an integer, maybe it'll say something. Or maybe the other way around, maybe.

It would say something.

Oh, you gave me the wrong type. Try using cast. Well, if it turns out that the person gave you the wrong type because they have a more fundamental logic error that's leading them to pass a truly wrong thing, CAST won't help them, and telling them to try using CAST is a bad idea.

Yeah, but that's why autosuggest, if correctly presented, doesn't say this is the function you definitively want. It merely says these are some things you might like to know.

Right, but note that also, as far as I can tell, nobody uses the auto suggest. Which auto suggest, like the suggestions bar.

Oh, no, no, I mean, I mean when you're typing. Oh, oh, right. I mean, you know, it'll Produce lots of irrelevant things when you're typing, but it's. Okay.

Well, the closest analog, we think Linters. Linters often do this and some in some languages, they have really nice splinters where, you know, as you're typing, it sort of have all these suggestions that you, you know, usually want to take.

Yeah, well, look, anyway, not. Not for this meeting, but can we. Can we get that. Presumably it should be in a user assistance meeting, this discussion, but it has to be a crossover between user assistance and error handling. Can somebody who's. Who's project managing.

Yeah, me.

I will pass it along and check that.

Okay.

All right, let's. Let's get into. Do I need to look at these other sections quickly?

If you want.

Okay. So we've got these working, S3 and Dropbox. Is that correct?

Yep.

Okay, good stuff. So what's the persistency of that? If I save the notebook and I have S3 credentials saved somewhere, and then I send it to somebody else, what happens?

That's all controlled through the new authentication design, which is in ref. Authentication.

Okay.

Okay. And what about when it's a local file on. On your computer? Are we giving any indication that that's not going to transfer?

I haven't really played with local files too much for this yet. I'm mainly focusing on remote resources.

Okay. Okay. All right, let's look at this. Are you.

Do you.

But how is that any different from any other program which references a local file?

Well, it's no different than.

How is it different from Open read every time?

Yeah, Open Read is more. You know. You know what you're doing. Look, video is driving me crazy every time.

I think that's different. I mean, so here let's say that there is an import at the top of your notebook in the file case. Okay. So we have all kinds of other use cases where we use import from a file.

Sure.

So I think that those are similar.

Look, when you're explicitly calling import or open read or whatever, I don't have a problem. The problem that I have is when there's a thing just sitting there in your notebook like a video.

But I don't think that we want data connection primarily to be for low. I mean, it's for data stores, not your local. I mean, so those are centralized.

I understand that, but I don't know what it looks like. Okay. Aren't there cases where tabular can be backed by something in a local file? There are, aren't there?

No.

Yes.

Data connection object creates a cached file. When I get a remote resource SQLite as well.

But that's cache. That's cache.

Right, that's cache. A backed thing would be out of court.

Right.

Say it again.

If we're talking about some file batch thing, that would be just kind of an implementation detail to an out of core thing.

Yeah, I understand, but if the file is on your computer, it's one thing if the file has a URL. Okay, let me ask this question. I mean, the. Somehow, okay, if it's a file colon thing that's sitting on your computer, the thing doesn't have a hope of anybody being able to find that if you send them the notebook. Right, but, but if it's a, if it's an ordinary URL, you know, to a global resource, then it does have a hope. It may not have authentication, but at least knows where it's supposed to go. Right, so what I'm asking is, and this is what's happening to me with video all the time, is I make the thing on my local computer, I send the notebook to somebody, or somebody sends it to me and they don't have the, you know, the video, the file backing for that video. I don't have it and they didn't know that I wasn't going to get it. Do you see what I'm saying?

Yeah, but I don't think it's the same thing here.

Why? I mean, it's file backing of data.

No, I mean, so the data connections is for data stores that are not on your local system in some centralized place.

But what if they're sqlite, which is a file on your local system? Yeah, I mean, they perfectly well could be some of them, not all of them. I mean, you're not going to have a big database server on, on your local system that doesn't have a reachable URL from elsewhere or potentially reachable.

So I think most cases of, of. Of a large database are going to be situations where whether or not people can access it from the outside really depends because you're not going to put a database endpoint on the wide Internet. Maybe it's in your company's intranet.

Yes, but I understand, but my point is that it is a uri. It's a universal resource identifier, as opposed to file colon, which typically only references your local system. Or a file object. Am I right or am I confused?

I mean, we have the file wrapper and the URL wrapper and there are cases where the URL wrapper clearly points to something like local host and cases where the file might Point to something that is actually a shared network resource.

So.

I regularly have files that point to, you know, the home, which is, you know, the NFS mounted home directory.

Okay, do you think I am confused and that we should ignore this issue?

I think that you're not confused, but I think that this is an issue that is pretty much orthogonal to tabular in that in general there is this issue that if you have a notebook with the pointer to a file or to a URL, et cetera, that's again for user assistance to have something, when you save it, say, by the way, know that this will not necessarily translate to somebody else's machine.

I agree that this is largely orthogonal and video is a slightly different issue. Because video, there's a thing that you would want the data for even if you're not rerunning the notebook, which is, oh, I have an inline video and I'd like to actually play the video. So but you know, for any other use case, you know, what you would have to do is you would have to kind of refactor how you evaluated the thing in the first place.

Let me push back on that for a second. Dataset, for example, has this issue when you're trying to page through data set. It has this issue, yes, like radio.

And the point is, if you want to deal with it properly in any venue, you have to refactor a bit how you've even authored the notebook. I mean, the data set or whatever may have started as an import, but that import isn't going to do any good elsewhere. You could load all the data into that cell, but then you have to use that cell as an input. You have to, you know, it's. You started with one flow and then you have to go back and reflow it, you know, to take that output cell and feed it as an input into something. You see what I'm saying?

Yeah, I do. It makes me think, and this is not for this meeting, that we should consider having something in the header of the notebook that's like the dirty flag, okay. That indicates that this notebook contains references to resources that will not, are not, you know, are not purely. I mean like, like we indicate which kernel the notebook is talking to, right? I mean when it's not the local kernel and it's kind of like it's indicating that it is using in implicit places like data set, like video, like I think future tabular with backings, with different kinds of backings that this notebook references, that this notebooks non evaluation capabilities won't work. Reference things that are not locally available. I mean, that are not included with the notebook. See what I'm saying? And I would. I would argue that we. I mean, we. Let's take this.

We have talked about that sort of thing before and I'm certainly not opposed to that. But one problem is that we don't. The front end doesn't have a very good way of systematically interrogating.

I know.

Boxes.

I know, I know, I know. So we have to. We'll have to have some registration mechanism which. We have other ones.

Yeah.

You know, just like this new symbol registration mechanism, so to speak.

And then I'll say one more. I don't want to go too far on this topic, but I'll say one more thing, which is that, you know, compound notebooks are coming. They won't be ready in the next release, but they are coming and they're. They will provide an interesting new set of ways to solve these sorts of problems.

A new degree of freedom. It's not a solution necess. I mean, you still have to put the thing into the compound notebook.

Exactly.

So, you know, it's a. But, okay, let's take this to user Assistance and, And discuss it. I mean, I think that would be, you know, like percent modernization. That's kind of one of these things. That's a, you know, a fairly broad, broad thing to fix. Yeah, it's a bit similar to percent modernization, actually. Okay, let's move on. There's a demo notebook of import. Okay, so we're about to see that. Yeah. Are you sending me that or how's. It's.

It's below. It's.

I guess it's included in the same. Okay.

Yeah.

These are the functions we have discussed most.

So how broad is the support for element wise?

That would be a question for Rob. I know it works outside tabular. I mean, can be used outside tabular, but I don't know how far it reaches right now.

Okay. All right. Okay.

To me, this is related to some sort of map. Sorry, operator, form of map thread. It would be nice to understand that connection.

Yes. Let's maybe take that to an LCC meeting.

Yep.

So this is the last thing that we were discussing. So some of the functions listed in the first bullet point are already there. And then handling of missing is something that we need to continue discussing with Julia.

Okay.

Particularly how to take a whole column and operate on the column, not just element wise, as we do with missing fullback.

There's more functionality here.

Is this being shown in this. In this agenda, or should we. I just Want to understand this agenda?

So the agenda has the summary of reports and then independent material.

Okay, so let's go to the independent material. Is there more stuff typesetting? Do I have to look at things here?

Well, those are the status. JSON is here.

Yeah, I could.

I'm prepared to show a notebook showing off some of the features of the typesetting, if people are interested. But there's a summary there.

Sure. Can you, can you grab the share? Okay, go ahead. Go ahead and request the share from me. Are you able to do that?

We got one button on Linux and it's. It just says share. Oh, here we go. Okay, it's working. Thank you. Okay, so here is. I'll do a quick summary of some of.

I'm not seeing this yet. You need to request the share from me.

He. Yeah, yeah, Both of you are sharing right now.

Oh, okay. So.

So you just need to stop sharing, Stephen.

Okay. All right. Okay.

But Jason, make sure to keep sharing until Steven takes it back.

Gotcha.

Okay, I'm not seeing this. Should I be seeing this?

It's in the tab at the top.

Yeah.

Or have. Have you.

Actually, that's it. Okay.

Okay.

If everyone can see that. So this is just a brief summary of the different typesetting methods used for each column.

Could you make it not so big? Could you reduce the size just so we can get a bit more on the screen?

How's that? I can make it smaller.

One more. Well, no, that's fine. That's fine.

Okay, so some of the features we have.

What's that hash. Weird thing, the top there?

This is our icon for numerical real type columns.

So I expect you to have a lot of opinions about this, the size.

Wait a minute, Wait a minute. So this is a new thing that's being introduced by virtue of the fact that we're typesetting these. These forms. Is that correct?

Yep.

Yeah, I don't like it.

So the point to distinguish if you have a string column versus a numeric column to. So that you can see that the type setting.

That's exactly correct. And a few others.

I. Profound. I think it should be a mouse over of some kind or some kind of info button. But it absolutely shouldn't spray. Spray this stuff over, you know, by default. In the, in the. In what we see. Am I making sense? I think it's fine to have something which is some kind of indication, but it shouldn't be anything like that big or that, you know, maybe conceivably something tucked in the corner. You know what I'M saying at like a quarter that size. What does the subscript blob mean anyway?

Just.

No, that. That's a decimal point. It's the. The idea is these are floating point numbers as opposed to integers which are. We can see below.

You know what? I think we're going Egyptian here. The ancient. I mean it's.

I mean I don't like the design or placement, all that stuff. But in terms of functionality, I can see it's useful. Yeah, like strings and categorical is hugely different than what you can do with them.

No, I think it's useful. I think we should tuck it in the right corner. Do we have somebody from design?

Jason, do you have. I mean these are just single column examples. Do you have. Can you show something that's multi column with different types?

Presumably you could do it.

Or just evaluate one of the tabular context examples.

Oh yeah, sure. Not sure if I have it in context. Oh, that one has.

Okay, okay, hold on, hold on. Let's look at this one. Okay, let's look at this. I understand. So can you scroll it over please? Okay. All right. Okay, so. Okay, look.

So. So this was all floats except the right column.

Yeah. Okay, so I've got multiple points here. Okay. The default width is. Is too narrow. I don't see why we shouldn't use the whole window width by default.

I was actually going to ask. I mean we do this data set too and it always frustrates me that we have the like. I'm not sure why it needs horizontal scrolling at all. Because if it has like a ton of rows, you don't at least I don't usually compare columns super wide. Sorry, columns.

Yeah, so do annoying if it's really tall. But do you want something that live resizes with the window?

I think that would be nice.

I'm not convinced at that.

But I'm not convinced either.

I meant more goes off the side and you have to use the windows horizontal scroll bar.

I'm sorry, you're advocating for that as.

Opposed to an NS scroll bar?

That's horrible.

Well, as opposed to it being.

What Christopher is advocating on. On the other hand, will make any line of text overflow because a very large tabular is going to make the whole notebook very wide.

No, it's not.

No, no, no.

Christopher is saying. Oh, I see what you're saying. Oh, that won't work. Yeah, that won't work because of the fact is that we're making the effective notebook width is as wide as. Unless we limit the text size. I don't think that's a. But look, here's what I would do. I would have the tabular, by default, be as wide as the notebook. Am I making sense? Whether it resizes dynamically. When you resize, the notebook is not glittering. Look, I don't think you should be taking the notes here, Jason. We've got notes being taken. This is being recorded. You don't have to take notes in real time, though I do appreciate the. The instinct to make sure you get things done.

So I. I mean, I'm. I'm really not convinced by that. And, you know, I mean, it's. Jason, did you. Is this window maximized on your screen?

No, it's. I can make it wider.

Well, no, no, but. But I'm just asking, you know, is this a typical default width window or did you resize it?

Yeah, this is. This is like rough notebook width on.

Okay, so there's no reason why the tabular should be narrow relative to the default notebook width. That's definitively wrong.

Okay, Yeah. I mean, there. There are separate questions here. One is compared to the default notebook width, and the other is compared to the current notebook width.

Okay, so look, look, my minimal suggestion is a tabular should, by default, launch itself at the width the current width of the notebook. Whether it dynamically resizes with the notebook is less clear to me. Okay, but, I mean, clearly resize that tabular, but by. By pulling on its. Its. Some. It's. Make it wider, right? Wider. Wider. Wider.

In general, though. I mean, there's no. I mean, all the space to the right of the tabular is just free. I mean, there's nothing there.

Yeah. Now, why did that do that? Why did it. When you pull it wider than the tabular content, why doesn't it just jam at that point?

So I do have that kind of solved. I just don't have it in prototype.

I don't think jamming is the right thing to do.

Nice to see that there's nothing more.

So the issue here is that you also want to be able to resize the individual column sometimes. And what happens when you do that? And I think I've made a suggestion to Jason, which I'm not sure if that's what you're saying you've come close to implementing, but I suggested basically an algorithm by which once you've gone past the natural width of all the columns, that basically you do a proportionate distribution of the extra space into the columns.

That'd be wonderful. I mean, otherwise, if you have to leave empty space, it should Be gray, not white.

Yeah, but, but, but, but I think, I think adding the space proportionately to the columns is probably the thing that's going to seem natural.

I think it'd be lovely.

What about. I mean, I, I've had multiple times to resize individual columns because the content wasn't fitting at the. And there is one thing that Excel has had since, I don't know, the 80s probably, which is like you can click, double click, not double click, but I mean, it's somewhere in a menu. But you can make all the columns fit the exact width, no more, no less than what it's needed.

Well, double click on any individual divider does it for that divider and if.

You highlight them and double click it does it for all of them.

That's neat. I always went for menus, but. Yeah, but anyway, are we any. Is there any chance that we'll be able to do something?

Oh, sure, sure. Absolutely.

Absolutely.

Okay, so listen, listen, there's a, there's a category of thing here which is how to distribute the available horizontal space. This example is not a very good one for that because there will be ones where there's all kinds of flowing text that has to wrap, et cetera, et cetera, et cetera. Let me understand the dividers here. Can you pick up a divider and move it? Yes. Nice.

Yeah.

And does that have a highlight that indicates that's going to be possible? I think it should. When, when you, when you.

Over the mouse cursor, you're getting feedback.

On the mouse cursor. I'm good, yeah, it's fine.

Can you interactively change the alignment?

Oh, well, I mean, I mean, in principle one could imagine a context menu that does that, but.

Does a data set have that.

I'm not aware.

Not alignment. No.

Okay, all right. But I mean all the other things like sorting and so on by that column, how does that work? Or does it.

I mean, there's a big question of what does sorting even mean?

Yeah, okay, fine.

That is a pretty non trivial question actually.

What do you mean by what, what do you want?

Like I.

Why don't you just run sort by like. I don't understand.

Doesn't. You know, we not take this, okay, can we take this tabular and make it a data set, please, and just see what it does. I think you can just say data set of that. No, no, no, no, no, no, no, no, no, no, no, no, no. Leave the tabular, leave the tabular.

Data set of percent.

No, this will be like from the, from tabular no, the data set directly on it will not.

Okay, so say from tabular. From tabular, please. And then of. I don't know whether it's percent 11.

Yes.

Comma, data set.

There it is.

Oh, sorry.

Comma, comma, data set.

Okay, nice. All right. Okay, so hold on. Let's just. Can we please just. Okay, can you scroll down to see whether there's a resize? There isn't on the data set, is there?

No, but notice how much nicer the number formatting is in what? In tabular.

Yes, it's correctly aligned. Okay, so let's look at data set. Go to the top of one of those columns, please. Now, I can say I like that.

I like the fact that data set is a lot lighter weight.

Yes.

So with the right mouse, you can have a menu and sort.

Okay, so that's the kind of thing that we're looking for. What's that?

I said, my God, I had no idea you could do that. But also, I'm not sure exactly why I'd want to do that.

Now, how does that change the data set, though?

Yeah, that's the question. What is percent 13? Now, is it sorted or not?

No, not percent 13. Evaluate that as an input and then does it preserve the sorting? And if so, how did it do so? Just evaluate, shift, enter on that.

Yeah.

Oh, it does not.

No, this is just for the view. Changes the view, but the data is not changed.

And so then this goes back to tabular. If you want to do tabular sorting, which I think is a reasonable request. But is it just a view operation? And if so, how does such a view operation even happen? How does it happen if you're pulling in data from a Hadoop cluster?

Yeah.

There are serious questions that are not obvious how you answer.

No, it's not obvious. It's not obvious at all. But for a small tabular, I mean, this is the point that. Okay.

And so this makes me wonder if we need to. In terms of the interfacery, that it might look different based on whether it's in Core or out of Core. You get some interface options in Core that simply do not exist out of Core.

Well, it's like what we have for these tabs at the bottom where it's, you know, this thing is locally stored, this thing is not locally stored, and so on. But can you scroll back up to the, to the tabular?

But, but, but it's, it's not that. Because this actually changes the very interfaces being provided to you.

I understand.

You know, the out of Core stuff is Just what would happen for shift enter.

Okay, look, I've got some comments here. Okay. I think that column at the left should be much narrower. I think it's only that width. The, the column, the 1, 2, 3, 4, 5 make that narrower and lighter, I think. I mean, it's only that size because, you know, early spreadsheets like that was just a, you know, a cell on a, on a what, what are they called? You know, on a. A terminal display, basically. I also think that you need to look at the, you know, look, look at the lightness of the gray on the data set that should be copied on the.

I, I'm. By the way, I'm not totally convinced at your, the suggestion you just made, but I'll just register my concern and move on.

Okay, but, but, but.

Yeah, for a, for a case in which there are many rows, I mean, the width of the first column might need to take that into account.

I, and, and I think that if, if, if you scrunch it in so that suddenly it's squeezed horizontally, but it's not squeezed vertically, it's going to look very weird.

Okay, well, I don't know, maybe we need those numbers lighter or something. I'm not sure.

But, but yeah, I would argue that the numbers on the left should be lighter just because they're kind of the most. The time, the least useful piece of information on the screen. And right now they have the most weight.

Yeah. Right. Jeremy, did you have any comments here?

I can't tell from the magnification, but they look too big too. Looks like they could be a smaller font size.

All right. Okay, why don't you guys. Jeremy, why don't you.

There is also an alignment issue which is just a bug that should be fixed. Jason, are you noticing the alignment issue?

Yeah, headers are aligned just at the center, but yeah, it should be aligned at the baseline for rows.

Okay. All right, so with respect to those, the indicators of what type of thing it is, I think we need to try some designs where we tuck that in the, in the top right hand corner. Lighter, but tucked in the top right hand corner. Is it. So I'm saying. Jeremy, Jeremy, were you here when we were discussing this?

No, I just got in. I'm not sure what that is.

Okay. It's an indication of what type of thing is in that. Is in that column.

So, Jason, scroll over to the right just so. Jason, just so that Jeremy can see the quote. So that's the string column. The hash dot is a floating point column.

Yeah. Okay, so I mean, we just need to. I mean, I really don't like the hash because obviously we use that in our language. Right. So I don't think that's the right thing to use here.

Well, and maybe more importantly, we fairly deliberately use a squared off hash in our language. I kind of like the idea of a hash, but maybe it shouldn't be a squared off hash that looks like a quad.

Yeah, definitely shouldn't be. Okay, so. But let's leave that to. I mean, you know, let's. Let's leave that. Let's get the complete inventory and let's let Jeremy and. And the team there try and come up with something. Makes sense.

Also, if you scroll down, you'll see them right now. Did you put them up separately, Jason?

Yeah, they're within some of the examples below here. So numbers, integers is like the hash without the dot.

Yeah, yeah, right. It's no good. Okay, but we don't even. So there's no point in. In showing it over and over. What the hell are those commas doing? The. The what We've got. We've got a way to display large numbers, and it's not that. So can we get rid of that, please? Use the way that we display large numbers. We're not trying to, you know, let's not reinvent everything here. Okay.

Yeah, yeah, for sure.

Yeah.

You can adjust that. They're shown with this. This example is showing that we adjust the column width based on the number length.

So I will point out that those are large integers. So you put, you mean the spaces.

Instead of the columns.

We have a way of doing this. We invented it 25 years ago. There's nothing wrong with it. Let's not reinvent it.

For larger, Larger numbers. We consistently use scientific notation throughout the column. So there's no numbers that are, you know, being displayed without the. Without the 10 to the power of whatever.

It's not obvious that that's correct to me.

We have.

We have a whole. I mean, we could. Yes, we. I think we could. Brett, can supply, and this is what we've developed elsewhere, what that heuristic is. And we have extensive test data if you want to sort of dig into it.

Right. Well, let's find out what. What is proposed here.

What if that integer is representing a uuid? I mean, it might. Might as well be the case that somebody. I mean, it's common to store UUIDs as integers, but you almost never want to display them as integers.

Yeah, that's a really good point.

Well, I mean, if it's a uuid.

Shouldn'T it be a string? I mean, it's not even a number really.

No, sometimes there are numbers, but.

But certainly the design goal is that, you know, you can change how any particular, you can override how a particular column displays, you know, and hopefully you.

Can do that right click menu.

I don't know if that will be true or not because I don't know if the overrides will be kind of as simply selectable as that. But yeah, I mean, you know, the defaults are important and so.

But John, I think that this is less controversial than sorting. Like if you have integers, show them as, you know, decimal, scientific hex or whatever is not changing the underlying data, so why not?

Yeah, I, I would also ask. I mean, in the top level right now we do scientific notation for. For reals, but we don't do it for integers. Integers just get really long and frankly, I'm not sure when you'd have a situation where you'd have really truly massive integers that, that really break things. Like, I don't know, I think it would be more useful to have like a dot, dot, dot in the middle of the integer.

I agree.

Than it would.

I think this is somebody trying to be a little bit too clever by half. I don't think we should do this.

Wait, you don't think we should ever do this by default?

Not by default.

Surely there are lots of data sets where the scientific notation is exactly what you want to see.

I'm not sure. But listen, populations for countries. Yeah. Are those integers usually?

They better be.

Down here. I've got them as a quantity.

Right, but, but honestly, I mean, something like that though is, you know.

Okay, listen, it's not that long. Okay, hold on, hold on. Is this the primary subject of our meeting today? Because. No, there's a lot of stuff in here and it seems to me. Okay, let's do a quick. Let's give some quick reactions here and then let's go on to our other topics. Okay, so, so Jason, can you scroll up please a bit? Yep, scroll up. Okay. Okay, so ahoy on our live stream is commenting. No, no, no, back down to what you were showing before.

Okay.

It's commenting that if there are missings and other kind of imperfections in a column, we should so indicate in that header. Do you understand?

Yeah, well, we've, we've already been talking about missing indicators.

Right. Some kind of X clam or something like this. Yeah, yeah, please. What are you showing? Yeah, go ahead.

So we do do custom type setting for missing but in the, in the column itself, not in the header. Oh, oh, products.

Columns.

My bad. Oh, and that should be just so far.

What should it be? You don't need to run it. What should it be?

Traditional form of missing. So.

Which is just a dash.

Just a line.

Okay. So what we should indicate is, you know, in the header and I don't know what the iconography should be and how it relates to that line. We should indicate that there are missings or other things that are funky. Possibly.

Yeah.

I mean, do we necessarily know, Right. If it's out of core?

We certainly. Well, then we don't know. Then we can't show if it's out of core.

You're not showing a table, probably.

Well, I mean, there's a level of information you want to show in the header and there's a level of information that you might want to show, say in a hover state or something like that.

Exactly.

Yeah. And this, this and, and the funkiness indicator might be in a hover state.

Possibly. I think there should be some indication.

Without hover or, or, or, or maybe not funkiness. Binary, but funkiness. Y. Funkiness. Details.

No details should be in the hover. Yeah, but just there is funkiness with little Xclam, I think is probably the right thing. And I think this would look pretty decent if this thing. If this indicator of the column character was tucked in the top right corner. Jeremy. Okay, let's go on and see some more Jason, please.

So for date columns.

Okay, So I just want to flag that. I think the integer thing is complicated and probably wrong right now. Okay.

Okay. For D columns, we'll automatically, automatically determine how the dates will be formatted based on the information in the column. So we have four columns here, each constructed with a different granularity. And then the typesetting.

Why the hell are the dates that way around? I'm learning what country you're in. Jason, are you in. Are you. Where, where?

Canada.

Okay.

I don't think anywhere day regularly.

How did they get to be this way around?

This is using.

It's one of the standard formats.

Yeah, but is that, is that based on geolocation?

No, this is how wired by default.

It's.

It's one of the ISO formats.

Okay. Okay. That's a separate issue. But I mean, 1984 Oct is not really anybody's version of that. That may be ISO, but that's not what we're going for here.

No. So this is, this is trying to take into account the, the, the how wide the spread of Dates is. And to decide the granularity.

I understand, but I think it's Steven's.

Objecting to the order. Why is it 1984 Oct rather than off 1984?

I think it's far more useful to be. To be 1984 first. Like, it's a lot easier to scan down the, you know, most significant digits.

Right.

Why is it that way? We've never had a format that looks this way. Do we?

I mean, this can be controlled, but. Yes. This is trying to follow the internal data list and more standard ISO formats, which are always trying to start with.

Yeah, it's not obvious. This is right to me. Can we please.

I really like the day. I mean, that's unambiguous, irrespective of what. What region you're in.

I don't. I'm not sure. Can we please flag this to talk about another time?

Yep.

Okay. We're probably going to want to have a meeting specifically about formatting. Okay. So keep going.

Okay.

You can also, when you're constructing the column, specify a date format string. In that case, type setting. We'll use that. So date, short ordinal date.

But you can see what those are no longer aligning, typically.

Yeah, I. I understand. So these are issues. We'll. We'll address them when we really study formatting. Okay. Let's keep going, please.

Okay. Quantity columns. So these are.

Yeah, it's a nice table.

Quantity.

Yeah, I understand. That's a nice concept.

String columns. We automatically calculate the column width based on the max length string. But if there's a really long string.

Yeah. What do we do? If there's a really long string. Yeah.

So it's just.

What do we do?

We have a kind of like a cutoff if it's longer than 24. So the maximum column width we typically use is 24 EMS.

But nevertheless, if you spread the whole, you know, we're going to need a priority for how to assign horizontal space to columns as we spread the whole tabular. Do you see what I'm saying? Yep. So that's a John. That's. That's something. Is that for you or who's that for? To figure that out?

So, I mean, so far we've been relying on, I think Brett and some other folks in terms of being able to. Because part of the problem is we might also be dealing with columns where you can't actually get all of the data for the column because.

Yeah.

So Brett's put together, like, sampling functions for us.

Okay. Okay. But. I mean, but we. We have to know if we have a choice between a long integer, you know, as we spread the width. Right. One thing we can do is spread the width equally. But should we be giving higher priority to finishing a word than finishing a long integer, for example? You see what I'm saying?

I, I see exactly what you're saying. And I mean, I have opinions about that, but this, but that should probably be for later.

Okay, fine. So let's please make sure that's in the, in the meeting about formatting as well. But if somebody could think about it before that, it would be nice.

Yeah.

Okay, keep going.

And I think the really last interesting thing to show is just that currently the way it's set up when you. The typeset form of the tabular can be interpreted as a tabular object.

That's lovely. Okay, column labels, please.

And then it just we. When you construct the tabular, it grabs the column names and throws them up as labels. And you can do it for extended key as well.

Okay. And I mean, there's some, there's some minor issues here with the, with what those gray levels are. I also would like to ask the question, should the right scroll bar be full width? I don't think so. What do you mean that, that right scroll bar of the tabular is a full width window width, you know, the same width as the, as the scroll bar for a standard window. Is that right?

Well, it's the standard width of, you know, of a pane. I think, you know, of an internal scroll bar. I don't think it's necessarily the same thing as the window.

Okay, but do we think it's right?

Well, I mean, as ITAI is saying, we're using it because of standard. That's, that's why, you know, it was already there for other things.

Okay. Okay. I mean, there's a bunch of silliness here. Look like the fact that the gray level of the above the scroll bar is lower than the gray level inside the top line in the tabular. You see what I'm saying? Oh, gray level around the arrow.

I'm sorry, this is just using the system scroll bar.

So there's things that are wrong there.

These scroll bars will not look the same from platform to platform is what Jason's saying.

Yep.

Okay. But we need to deal with that in some reasonable way. Okay. Do you want to scroll to the bottom of this notebook?

Yeah. There's just one thing that you can do. This is this should span one day. But that, that was just a small thing.

And that's all I seems nice. Okay, hold on. Wait, wait.

Steven is.

Wait a second. Okay, you can go ahead and stop sharing. Okay. All right. Is there anything else we need to see? What's this visualization stuff?

Right, so.

So this is the plan that we're following for viz. And we've sort of grouped functionality into different levels.

And level zero is no special tabular.

Support at all, though entirely on you. I understand that's not very useful. So level one is what have you implemented so far? What have you implemented so far?

So right now we have level one supported for pretty much anything that has.

List and plot in the name.

Okay.

Yeah.

So list plot, 3D, list density, list plots. Yeah, okay, I got it.

Yeah.

Etc. Next up are charts and some of the statvis, you know, sorts of things. And part of level one is making.

Use automatically making use of the type information from tabular. Yep. Okay.

And yeah, that.

Is there anything you talk about here for this? Not immediately, no. Okay. All right. Are we on to the PIOTRS IO material? Is that right?

You don't need to open init. I. The. The way I set it up, there are some large files used. So, Stephen, you don't have it, but all the output is stored. So it just shows the timing and memory used for large data sets. I want to demonstrate here that this is really efficient. So this shows the file size more than 21 gigabytes in 36 seconds. You can get tabular, and that requires 26.

So the tabular that's here, is that an incore tabular or is that.

Yes, yes, that's incor.

Okay.

And you see the byte count of that expression? Okay, almost 26 gigabytes. And so this is getting everything from the large file, but we also have the partial import. You can select columns or you can select rows. And that's the syntax.

So this is rows and this is columns.

Yes.

Okay.

Do the columns have to be integers and not. You can't give the name of the columns.

Well, it's a CSV. It's not going to know.

Yeah, well, we get to that. No, it's in this partial access import. You on. You can give span instead of list of integer, but it's not column names.

Okay.

And in this next example, it shows the selection of rows. So this file has a large number of rows.

Adino actually does rows. Does row one. Is that the first row after the header? If there is a header.

Is that right for a CSV? I mean, what if there are three layers of headers in a CSV?

So, yes. So here in this partial access, we are not counting headers.

That's what I thought. In other words, the first row, the first line of the CSV file is the first row correct?

Well, the first row of data.

I'm sorry, what?

So, okay, so something you. I think the tabular CSV importer. Correct me if I'm wrong, Piotr, but I think if you just say import of a CSV and CSV has a header, it creates a tabular with the appropriate column names. Is that correct?

For CSV, the default element is still data. I don't think we should change that.

No, no, no.

If you ask for tabular, if you ask for.

It has header detection and it will use them as column names.

Hold on, hold on, hold on, hold on, hold on, hold on, hold on. What if there are three levels of strings followed by the rest of the file as integers? Does it deal with that? I would doubt it. How many lines of header does it allow?

Well, we only choose one row to be a header.

And what if there's no header? What if it's all numbers all the way down?

Yeah. So this is a discussion below. We have to discuss whether we should generate the headers or not. We currently generate column names, basically, if there are no header in the file.

Right, but you detect that there's no headers through, you know, Apache Arrow, Magic or something or.

Well, yeah, we. We have efficient way to. To. To discover that. And you know, most of the time it's.

Right, so, but what if the. I mean, so for example, if the whole file is strings, you have no idea whether the first row of strings.

Yes, exactly. If the. Each column is a string, then then there is no way. So you basically say that the first line is also a data, it's not a header.

Okay, fine, fine. So, I mean, it will.

If.

If everything seems to be of homogeneous type, it assumes that there is no header. If there is a distinct type that is more stringy, probably. Although I could imagine that it could also be more numbery at the top. Then for either one line or multiple lines, it will assume that those are headers. Is that correct?

Well, it may skip some lines until it decides that this one is a header. Because it could be either a title or something that doesn'. Match. You know, the commas don't match.

I see. So we'll literally skip it. What will it do?

Yes, yes, I have the. I have the example. Well, for Excel, but it's basically the same situation for csp.

Okay.

So here the. What we Call column types. We also call it schema. Right. Get the type of each column. And notice, even for such large file, it takes very. It's very fast and uses very little memory because the way we do it, we only look at the first 20 rows.

That's okay, but in this case you're looking at 10 million rows.

Yes, and that's showing. I mean, it's still 10 million, but. But the file has 160 million. So if, if you would.

Cool. Yeah, it's cool. Right. Okay. I'm not convinced about this option name. We should discuss that.

Sure.

Okay, can we please pin that as a thing to discuss?

All right, so column count, not much to talk about. It's. It's. It's fast and doesn't require much.

Okay, can we get a list of the things that you are the element types that you're proposing for CSV files.

Can you scroll up? I think it should be at the beginning elements.

I see. Okay. Is summary new?

No, it's actually present for all formats. It's a framework element.

Which of these are new.

Tabular column labels? Column types.

And what is the return for column types? What was that? What kinds of things?

Well, it's here. It's what we also call schema. It's an association column name goes to column type.

But what are the possible right hand side types?

Well, for CSV. Well, it could be. Well, again, I have the example. Not all types that we support in tabular can be directly supported in CSV. But the way we do it on export, we convert it to string and then we import that as a string type and that string is actually input form of the expression. So I'm sorry, this is essentially just.

A subset of tabular column types for csg.

Have we reviewed the tabular column types?

I know that we've looked at it a couple of times.

Yes, we have discussed them a few.

Times.

With missing data.

Right, right.

Can I ask a question about this thing about being a subset of tabular type? Like I'm expecting, like if you're importing integers from a CSV, it doesn't yet make sense to say whether integer 64 or integer 128 or whatever, they're just all integers.

Right. Well, so we relying on this arrow library to tell us and what could be. You're right, Carlo. So say that the first 20 rows are only small integers from 1 to 100. And then at the bottom of CSV file, there are large integers in that column. So the schema would lie by default because it May say that it's integer 8 instead of integer 64, which you actually imported all the rows.

Okay, so. Okay, but I'm still. Can we please make sure that we actually look at all the type designations for columns? Okay.

Yeah, for CSB or, Or.

No, for everything.

Okay.

And another thing I was wondering is like, if you have like, is there a concept of a threshold while you're importing something? Like, like I imagine you have a column where 90% of the things are numbers and then every once in a while there's a string that is actually indicating a missing. Can you set the threshold after which instead of trying to like it keeps importing it as an integer instead of a string?

Well, first off, you'd give a missing value pattern.

Yes.

So that those strings would be detected correctly as missings.

Yeah, I mean what I'm saying is like.

And either way, even if there are non conforming types, it should be an inert expression. So get all the data.

We have some heuristics for this. We have heuristics already, but some of that can be parameterized. But we can look into that.

Yeah.

Okay.

What I was thinking is that instead of saying this is the missing pattern, I want you to know that if there's like 99% are integers, the, the things that fall out of type are automatically missing. Can I set that 90% some way?

Not right now in import and probably not.

That sounds like cleaning.

Yeah.

Again, there is some of that.

But let's keep going. We've got a lot to go.

So here.

Steven, here is the important question that I would like to ask. What do we do?

Yeah, I was going to.

There are missing headers.

I. I got it. Right. But by the way, practical point, I push back my next meeting by 15 minutes. So we have 15 more minutes here.

All right.

So can tabular have integer column names?

Well, yes, actually. That's what happens when you create tabular without giving any.

That's what I was going to ask. We support you just create tabular from a matrix. It'll just be 1, 2, 3 for the column names. Yes, integers. 1, 2, 3. Right?

Yes. But can you actually use, use them in the processing functions? I. I'm not sure if we have tested that or if that works.

We should be able to.

I hope so. Yes. Instead of strings as a column names, you just give, you know, these consecutive integers and that should work.

That's what I think we should do. And I think they should be displayed pretty much. And to indicate their integers we could display them pretty Much like those left hand row indicators. See what I'm saying? By the way? I mean, it just is wrong in my view that the, that the row, you know, formatting wise, that those row things are in this boom, you know, heavy gray. Get what I'm saying? They're the most visible thing on the page, which is just wrong. I think what we've done for line numbers, for example, in the package editor is a more sensible way to do this. Am I making sense? Anybody? Jeremy? John? Anybody?

I'm looking now. I don't remember what the package editor looks like.

Okay, but, but John, do you agree?

I'm sorry, I was not engaged.

Okay, all right. Okay. So look, Jeremy, I just. Let's spec this so that these are the lightest thing on the page. Okay. And then what I'm suggesting is that when these are missing, they look just like. It look very much like the, the scaffold numbering on the left. You see what I'm saying? And by. In the.

Say that again.

Well, okay, so when the, when there are no explicit names of columns, okay. And we're just numbering them, one thing is we're now using integers as the designations of columns, not strings. So what I'm suggesting is that what we do is the thing looks kind of it. Like it has scaffolding numbers. Right? The numbers don't really. Aren't really significant. They're just scaffolding. They're just, you know, they're just sort of, for your information, type numbers here down the left. And I'm saying if these things at the top are just sequentially numbered because there was no explicit header, then we make them formatted the same kind of way as we're formatting the left.

Okay.

Yeah, that makes sense to people. Okay. And I think the names should be, should be consecutive integers, piotr.

So in a sense, no generation. Because that's what happens by default when you pass matrix to a tabular. And that's for me, I have to invent.

Don't invent something out of, you know, just sort of pull it out of the air.

That's fine.

Okay, so if missing one actually had a label, so that was foo or whatever, you know what then missing two.

Would still be just two. Two. Okay, But I doubt that that will happen in CSV because I think it will fail in that case and think that the label foo is just a. There's. There's no way of omitting that. There might be in Excel, for example, but in a CSV it will just think it's A. It's a title or something, won't it?

Definitely.

The comma separated value might. We might be able to get it. Go ahead.

We already have a precedence for this. If you do like Import of a CS file with data set and header lines goes to 1 and 14, any missing column name becomes missing the number which was given by you, Steven.

Humph.

But I worry about mixing like you said, if the first column is foo and the second is missing, the second is two, just the integer. How does that work when you use that in a pure function? Like you have foo, but hash two means a different thing. It's not hash string two.

So that's.

That's what I was asking.

It could be confusing if you mix indices and names.

And also the dash is really annoying if you're doing a hash thing because that syntax doesn't work.

And, and keep in mind that we are. We're not allowing the same column names, column column labels. So import. If. If you say you have the only the first row has a label and the other three don't, then only the two rows will be. Two columns will be imported. Because that's how tabular works. It does not import something that has repeated column label. Unless maybe Rope already added an option for that, but I think that's the default behavior.

Okay, look. Okay, we've got a bit of mess here. Okay, first point is, okay, you're saying that in the case where we're partially labeled, okay, what should we do for the other columns if nothing is labeled? The objection that Carlo raises about 1, 2 is an interesting objection. And I'm not sure how much of our processing functionality falls apart in the case where we are, you know, where we have numbered columns that we need to look at. Am I making sense? Does anybody know? Or is Rob the only person who knows that?

Well, I mean, when we only have numbers, right, we need to use a different syntax.

Right?

It's pound of part, not pound, pound part. Right, It's. Or it's really pound, bracket, bracket.

Well, I mean, this is something we have changed a few times. I know that by default matrices are not upgraded to associations with integer keys. We don't do that.

Yeah, and we shouldn't.

Right, Exactly. For the reasons we are discussing here.

But they are typesetted like that. Right. And there are these virtual consecutive integers.

Yes, yes, but it's not that. If you do normalization, then you get associations back.

Right? Yeah.

They're not actually column names, their positions.

That's correct. Which is why I think they should be done like the left hand side. See what I'm saying?

Yes, I agree with that. That's a good idea.

Exactly.

Yeah. Okay, all right, fine. So let's. What we're addressing is the. Okay, so in the case where it's purely unlabeled, that's what we should do. Now let's talk about the case of partial labeling, which seems a bit. I mean. So. Well, I mean, I think then if it's foo and then, you know, sort of default to. Then in order to address the default to column, you'd have to say of two, so to speak, part two. Correct. Even though for the first column you could say hash foo. Is that. Is that correct?

I'm not sure if it has to be given a string name or not, is my question at this point.

No, but I understand. But I mean, so, so my. But we can have an association that mixes string keys and non string keys, right? So we. We've clearly got to promote to an association. If anything is labeled, it's got to. The whole thing's got to be promoted to an association. Does that make sense?

Yeah, but I think in terms of usability, having to type hash part one to indicate that that's the key one and not the position one, it's like way worse than having an auto generated string that just says missing one, maybe without dash, but seems like.

Well, there's another possibility, which is to have the string 2. In other words, it's foo and then the string 2, the string 3. Of course that's not going to help you because the syntax 2 doesn't represent a string. Okay, I've got a better suggestion. Okay, rather than missing, which sounds like an error, which this really isn't, why don't we just call it column two, that's foo, column two, bar, column three, column four, whatever. Rather than kind of sticking your tongue out and saying, you know, you. You left it out.

The index to be at the position you'd want them consecutively.

No, I'm saying if, if it's foo. What I'm saying is it would go foo. Oops. And then it would go column two. But, you know, you see what I'm saying.

I like this proposal so much that I like it over even for the. Like, I would do it for even the case where no name is present because it's so much nicer to type hashcom1.

But I don't think we can do that because of the, you know, normal stuff.

Yeah, I think that we want to keep those as matrices, but I do really like this as well.

Okay, fine.

Could we have at least a two tabular directive that does this?

Yes.

Auto generate something names and then we could even decide like provide the prefix for instead of column, should I.

Change the missings in data set to column two, etc. So that they're consistent in existing. Probably limitations.

But let's look at that one more time. But probably.

Yes, I. I think we should.

Yeah. Okay. All right, fine.

Right. So. So one more time. If there is header missing. No, no, no labels. Then we leave it like that. We don't generate anything.

We only generate it and we would display it as a. As a kind of a. A scaffold only header.

How about an option as a string defaults to column and it always follows by its, you know, indexing. But then you can set it to whatever string you want and then you can get back your kind of missing dash if you really want to.

Sure. You can. You can give a prefix string and if the prefix string is none, then it would go to the number.

Yep.

And what should we call the option?

Well, we have column names option already in import of CSV. Right, right.

But this is a prefix for inserting auto generated columns.

So why does slightly different column name prefix.

Sure.

Okay.

Column name prefix as a. I want to look at that again though, that I'm doing this too.

I don't like column name prefix because it doesn't tell you that it's when it's missing. So I would expect it to.

How about missing column name prefix?

Yeah, let's just start with that. And changing names is easy.

Okay. But you got to remember to bring it up again. Okay, we will. I mean, we should review all those options.

Ok, so here, the rest of this section just shows that header detection works when. When there is a header in the CSV file.

It is going to be annoying to people that if the column name is tree species, that you can't say hash tree species because of the space. And I assume that it makes no sense for us to have some, you know, auto smooshing if it makes no sense.

I mean, they can do it. They just need to remember to include the quotes.

Yeah, I don't see the problem with using quotes in this case.

Well, but it will be hash part.

Tree species, hash quote, tree space, species quote.

Wait a second, what's the syntax? Hash quote. Does that work?

It does work, yes, if you spell species right.

Well, okay, but that's a different issue. I didn't know that worked.

That has worked from the beginning of hash. You know, it Was it was close to it?

Yeah, I mean there, there were issues early on, so it wasn't documented, but it's.

Is it documented now?

I, I, I'm pretty sure it is.

Why don't we track. I didn't know that worked. I had no idea that worked.

I would have never guessed.

Well, so you didn't know either. So, I mean.

No, I knew it worked.

I thought it was documented, but it looks like it might not be. At least not in the documentation for slot.

Okay. All right, so we need to fix that. That's great that that works. Does that change our mind about how we do column one? No, I don't think so. I think, I think the smoosh is confused. Convenient by default.

So there is one question here. If we actually start generating the column names, should we include them also in data and row data elements? They are not included right now. But they are included if they are present in the file.

Yes, that's fine. But if they're not present data and.

Raw data, just take the data from the file as is.

Yeah, right. It's only if you're tabulating them that. All right, we want to do this.

Okay. As long as we already.

I had no idea this worked. So some subset of people. Christopher knew this worked. I don't know what. I wonder if there's any indication.

Well, I, I knew it worked because I helped implement it.

So yeah.

Okay. When did you implement it? Do you remember?

Maybe five years ago or. I, I, I, I, it's been a long time.

I, I think it's more than five years ago. I think we released 10 without it. And that was 10 years ago, version 10. So. And this was like maybe 10.2. So I want to say eight years, But yeah, association is 10 now. Isn't that wild?

Very wild. Yeah. Not a moment too soon. We should have introduced it earlier, but it's nice thing. Okay, one question about this is autocompletes and so on in when you're doing operations inside tabulars, we should consider, we should discuss that because for example, indicating to people that you can do this using some kind of auto suggest when you're doing operations on a tabular would be really nice.

Hi.

I don't know how we route ourselves back to the tabular's keys.

Yeah, I don't know either. I just want to raise that question. Okay.

So we have the header lines option that, that's not new. But using header lines close to zero, we can disable header detection if something goes wrong. I mean, yeah, I understand.

Okay, I'm just going to scan this very quickly. We're going to have to come back to all. Sure.

So here the conversion only relies on the library, this RO library. But we may also, you know, start including ROP's code for. For doing column conversion directly in the input. But that's not done yet.

Okay.

And here is the missing value pattern option. So I came up with this list. Import, export, vectic, default, dollar default, missing value patterns.

That's pretty wild. Interesting.

But the last 10 or 15. This is work from language, but the rest is different.

Is it right that the various infs like infinities are missing and not some kind of infinity? I don't know how we do infinity here.

I think most of the tabular stuff doesn't support infinity.

Well, it could be the inert expression type. Right. Only that type supports it, I think. And Christopher, this is just a default. You can obviously overwrite it and if you.

So we imagine. We imagine we're actually having a dollar variable that contains all this weird stuff.

Well, that then maybe it should be moved to system. But that was my proposal.

Yeah.

I think it's mostly just live as an option. Right. Although maybe you'd want to be able to.

I think you want to be able to thank switch it. So I think having a dollar default missing value patterns is a reasonable thing to do. I just wonder, should they be regexes? Can they be regexes?

Not for import? Because we have to have list of strings and that's why those are just the default patterns.

But if you look at missing value pattern, that is a string expression.

Yeah, that's right.

When it's given to the option.

Yes, yes, yes. But I'm talking about that list of default values.

Why do these have to be literals and not.

The library doesn't expect any regex. It has to be list of strings. So if you want to rely on the library. That's the limitation.

We probably should talk about this some more.

But we do support basic value pattern goes to regular expression or string expression. That's what I'm demonstrating below.

Yes. Okay, well, it may be sufficient to have these. These defaults be. But so in other words, we can deal with this.

Right?

Yeah.

So here the negative number or empty string.

Okay, but I mean, so if we put one of those in here, all we're doing is combining whatever mechanism we have for dealing with a string expression, a string pattern for missing value pattern. We could also do it for these things here. Just that the default probably shouldn't contain the Default probably shouldn't contain anything.

No.

Okay, I see what Piotr is saying. This is unfortunately an implementation detail because there's a few engineering things that we did underneath the hood to make that string expression thing work.

And it's somewhat weird we have to import twice, basically. That's unfortunate. But that's okay.

Fine.

So. So the answer is the default should not contain any string patterns for now. But it's okay. But I mean, so that's fine. I mean that, that's a. I don't think we should disallow it. But you know, the default we provide will not contain them.

Yeah, sure.

And then if you add one, it's a major performance hit. But that's fine.

But that's okay. I mean, maybe one day that'll be fixed.

All right, so that's, I guess all I want to show on import. This is export.

Okay. There's a lot to see here. So I mean, how similar a lot.

Of these other tabular formulas. Parquet is a really kind of good modern tabular format to get basically the.

Most important format right now in the.

World of tabular, but also not much controversial to discuss here. It's just very good to have. By default it imports tabular and I.

Think these are compressing somewhat too.

Right, right. So I have the. What I show below here is the compression options and include meta information. They also can store meta information. But we have compression. Compression level.

We should talk about that as a global thing because when we have that for GIFs and things like this, and I don't know whether string pattern string option names are the right thing to do. Are you proposing a non string option name for the stuff?

Well, compression level already exists as a symbol, so that's what I reused even though I forgot to change it here. But it can be a well important.

Export has the weird Everything kind of looks like a string because every format has its own quote unquote options.

Okay, there's a lot of exciting stuff here.

Last thing. This, this, this xls. If you, you already opened that section. So we have enabled header detection when there is a single table in a sheet.

So that's what happens if there are multiple.

We have by default for now.

Well, it's not disabled, but we consider disabling that. But also we already have some preliminary results to try to recognize that case. It's not in the prototype build yet.

Okay, but just so I understand, I mean, you know, we've discussed, you know, people who put seven tables on a single sheet type thing and whether we.

Can detecting those separate ones and finding their positions and bringing them all into individual tabulars is something that I'm prototyping right now.

So you think we will have something on that.

With varying success. Okay, like what are we going to.

Do with the rest of the stuff that's on the sheet? Are we going to throw it away or are we going to have some way of bringing it back as some kind of. I mean, I think very uncharted earlier.

I have a suggestion.

I have a suggestion. Okay. My suggestion is that we have another named format that isn't xls that something like, I don't know, XLS sheet or something. I don't know what the right name for it is. That basically brings back a data structure that is probably an association by default that contains, you know, oh, you've got this in this region and that and that region picture here and so on.

I just suggested this morning to Piotr that we should have a new element that is an association of spans goes to the table that was in that span.

Right. I'm suggesting that maybe we want a new top level type, but you might be right. I, I think it'd be better to have a new top level type. I think it's easier for people to understand that. Now I'm bringing in a composite spreadsheet basically. I don't know, I'm not sure.

Is there any different granularities of elements in terms of like metadata by sheet or whatnot? There's partial access. So introducing another element that says this breaks down a sheet even further to subtables is very sensible.

Another element as opposed to another type, another import type. Is that what you're saying?

Yeah, this is just an element.

I understand that. I understand the question. I, I guess what I'm photo suggesting is that there might be another format type that has different defaulting about what it brings in.

I'm not quite sure what you mean.

Well, I mean xls, maybe it's default. Well, right now it's default import is data. Yeah, right. Which we might change to its default import being tabular maybe. I'm not sure if we can do that.

Less convinced for Excel because of how like spreadsheet and visual it is.

But.

As opposed to CSV, which is often columnar. Parquet, which is absolutely columnar and well.

Typed and yeah, parquet, it's obviously going to be a tabular by default.

Right, right. But, but so CSV, you're often going to end up with a tabular.

Yeah. And I don't think we can change it. Well, CSV, the question is, can we change it? And I'm not sure we can.

I. I'm not sure we can because of how well used it is. And I'm just not sure it's as sensible in Excel.

Fine. Fine. But so, so, okay, so you're arguing that the, that the span associations and so on, that the detailed information, the, you know, positioned, what we might call positioned import or something is just another element. And I get, I guess I'm sort of, I'm. That, that doesn't sound silly given the rest of the things we're talking about here. All right, we need to wrap this meeting up, but this is, this is looking good. Now let's see, we have another meeting this, this week, right? Yes, we do. On, on Thursday, right?

Yeah.

Okay. See you all then.

Okay.

Bye for now.

Thank you.