Live CEOing Ep 747: Language Design in the Wolfram Language [Longest + Shortest and More]

Everyone, welcome to another design review. This is partially part of our season of version 14 design reviews and one of four that we're doing today. So let's start off with the longest, shortest story.

Right, so this is something we discussed earlier this summer. And we, you know, so I. One of the things was that we should add some documentation clarifying the situation. So you should, you know, check out those. Those pages which are. Those are links. So that should work for you.

Right.

So that.

I can't remember the issue. Can you remind me the issue?

So. Right. So when people use. Well, I mean, one of the several.

Issues.

Is that, you know, longest and shortest sort of have different meaning when it comes to optional. When. And so when you. Right, so that's sort of what we've had meaning here. So rather than sort of literally meaning, you know, longest or shortest, you know, with longest at first attempts to match the pattern and only then uses the default if optional is used, You know. Right. So it's sort of, you know.

You.

Know, different from the normal, where sort of shortest is, you know, sort of the default.

And yes, the level of uncertainty that you're expressing in your voice and talking about this is disquieting. What is the. I mean, you know, in other words, I, you know. Okay, who else. Who. Sorry, I'm not. Did we make a change to the code here or is this just a documentation change?

This is just a documentation change. Right. And so if you. Right. Look at sort of one of the. Sort of the question that sort of sparked this whole investigation, that's the. Why does longest not help here?

And. It.

Right. If, if you want to.

I'm sorry. Functionality wise, we're not. I'm. This is all incredibly obscure. Let me, Let me just ask this question. We don't believe there's a problem with functionality. Correct. We don't believe there's a problem with functionality. This is pure documentation. If you guys think this documentation helps the situation, I'm sure it's fine. I don't think we need to discuss this here unless there's a question about the functionality.

Okay, so. All right, so there is sort of a bigger question about the functionality, which is, I mean, and there are a couple of, you know, sort of points about here. So originally, Optional required that there be, you know, a blank inside of it. You know, you know, not blank, sequence, not anything else. It had to, you know, it could be any variety of blank, but it had to be exactly one blank. This wasn't documented anywhere other than in an error message, but was it just.

An implementation limitation perhaps.

And.

Right. So now we.

Right.

But in particular this behavior of longest and shortest made sense because, you know, there were only two possible lengths, 0 and 1. And so the, these sort of meanings of longest and shortest sort of effectively made. It made the blank either not be used or be used. And that made sense. And the fact that it was. Had the default, you know, opposite default made sense because that, you know, it would be stupid if you always use the default and never the thing inside of it. You know, then we made this. Now we didn't sort of intentionally change this unless, you know, Danny has turned up evidence that he did this intentionally.

But.

Right. We did make this change in version 10 point something where, you know, the default value didn't have to match the same restrictions. As the, the pattern. Right. So you could have like a blank string, but the default value could be zero. And when he did that, he also allowed sort of more general patterns, but you know, they're sort of edge cases. So for example, you know, there's that bug listed there that, you know, if there isn't at least some variety of, you know, blank in there, optional, doesn't do what you think. And you know, Nick M has some, you know, very. In the last section there has some very, you know, obscure thing of using longest with alternatives. And I don't even, you know, we haven't defined what that means and we.

Didn'T have pattern sequence at the time when we invented longest.

Right. And so, you know, do we want to spend the time to define what these things mean? I don't know.

Right.

The second we don't right things for version 14. This is not, this is not a burning issue.

No, yes, I, I agree. The one thing is that we might, Right. The one thing that we might want to think about is whether the, you know, the, the requiring the blank is. Is that something, is that a bug of something other than P5 or is that.

Look, this needs somebody to dig in and you know, I think this is a, A when the pattern matcher is rewritten, we should rethink this.

Okay.

And I don't think it's a, you know, I, I just don't think this is a corner case of a corner case. It's all right. Yeah, I've been bitten by this because I've, you know, randomly used longest and shortest for things and I don't understand what they do in the context of triple blanks and things. But you know, I just keep on poking it until it works. And that's not Great. And I think when we rewrite the pattern matcher, we need to look at this more carefully. But I think anything that we try and do other than that is going to be a hack job because we haven't really dug in and understand the full context of it. And I think it's better not to do anything and just to add documentation.

Okay, well then yes. And since, which is, you know, a perfectly reasonable place that, you know, I, I was hoping to get to the main thing. I was, you know, that was the sort of level of decision I wanted. And you know, if you're okay with.

I think I'm okay with that. Committing in this new documentation to something that we're going to regret later.

I, I don't think we are.

Okay, fine. Okay. Trace original to automatic. So, I mean, I've been now using. Sorry, can I leave? Because I was, you were there for the first thing. Do you have anything?

The first thing? I mean, I'll be reading email.

So do you have anything to say about the first thing before we close it off?

No, I, I, I agree with everything.

That you were saying.

I mean, there is a detailed analysis.

And we can have simpler or more difficult examples, but I agree with what, what you decided.

I think that's fine. See you, Tom.

Okay.

Okay. I mean, isn't this an Ian production, this Trace original thing? No, it's. Ian is the user of this in Trace graph. Correct.

Right. So, so, I mean, Danny did the implementation, Ian is the consumer and they're both here and you know, we have some, some pretty examples there for you. You know.

This is perfectly reasonable.

Okay. And you know, if you're okay with the note that Ian.

I think I am. Yeah, this looks reasonable.

Okay.

Oh boy.

Yeah. So this John will need to take over here because I'm not sure I fully grok.

Yeah. Okay. So, well, first of all, you can open up the documentation for notebook import.

Which version? Let's see, where do I need to go?

It. I mean, this has been around for a while for a few released versions.

I just have to open generic notebook import. By the way, I found a horrible thing with, with them. I've got a RAFT failure again. Okay. I have notebook import. I get a raft. It's all good. I open the thing up to this now. Oh my gosh. The raft goes away. I'm about to type an argument which is exactly when I need the raft and Kaboomasaurus. The raft goes away.

That looks not a life raft.

It is a life raft.

This isn't a new newly Reported thing. You. You mentioned this in the last user assistance meeting. We talked about this.

Okay. All right.

This particular thing, I mean, I don't even necessarily think that's wrong. It's not. That's wrong.

Okay. All right, let's not cover this right now. Okay. Notebook import, okay?

So if you open up the details and options section, you'll see that we already have a thing. We have two options documented here. Interestingly, both of the options are in their string forms because they're not used elsewhere and I don't know, because people are being lazy or something. I don't know. So, okay, so flattened cell groups is already a thing that works for Notebook Import. I'm not proposing any changes for Notebook Import. What I am proposing is that flattened cell groups can become an interesting option for a couple of other functions for cells. It could do exact. Exactly what it does for Notebook Import right now, which is just to wrap nested lists around cell groups.

Nested lists or flattened. I don't understand.

Well, flattened cell groups is true by default. So by default, you get things flattened, but you can set it to fault false if you want. The cell groups.

By cell groups, we mean that it is a cell inside a cell. Like this.

Yes, yes.

I don't think it's really flattened cell groups. What I think it really is.

Sorry, sorry, sorry. Not a. Not a sorry. I don't actually know what you mean by a cell inside a cell. Yeah, yeah, but I mean, that is how we represent the serialization of cell groups.

I know. Yeah, I know. But what I'm saying is what flattened cell groups is doing is so taking this out from being a nested cell object and turning it into a nested list.

Well, take a look at that first output of notebook import where I've set flattened cell groups to false. If I hadn't set flatten cell groups, you would have just gotten a flat list because it's true by default. But by setting it to false, notice that it nests the. In this case, input and output cells together, which. Which are grouped.

I understand. In other words, it gives.

Okay, here it's just using lists for nesting just to be simple. I think, you know, because notebook import wanted to, you know, produce simple results.

As opposed to what?

Well, as opposed to what Notebook Read does, which is, you know, high fidelity, but requires high degree of knowledge of what's going on.

Okay, so let me just understand. The issue is if you use nested lists for cell groups, then the head group of a cell appears where.

The. That statement didn't make Sense the head. I don't know what you mean by head group of a cell.

Like the section cell, I think.

No, the section heading cell.

The head cell. The head cell of a group, you mean?

Yes.

Right.

Okay, so right now, cell group data. The cell group data contains the group of cells. Right. And the out side cell is the head cell. Is that not correct?

No, that is not correct. The cell group data contains everything, including the head cell.

What's the syntax of cell group data?

So cell group data takes two, two arguments. The first argument is a list of cell expressions, and then the second argument is the state of the cell. Is it open or closed or reverse closed or that sort of thing.

Okay, all right. So in the flattened cell groups.

But the thing is that cell group data never appears nakedly. Cell group data always appears inside of a cell expression.

Why?

Because that's the syntax that was agreed upon before I started working on the front end. As you know, that goes pretty far back. So I mean, I mean, the reasoning, I think, was back in the day that the first, you know, that you had postscript data and text data and box data and bitmap data, and all these were kind of different cell types. The idea was each thing was a different cell type. And so then the question was how to represent cell groups. And in hindsight, I don't fully agree with this decision, but we've been doing it forever, so I don't want to change it. We said, well, let's just call that cell group data. And like everything else, it would be wrapped by a cell, even though there isn't any actually a cell around it. It's only wrapped by a cell in the serialization.

Well, I mean, the serialization we can't change, I guess. And we have great compatibility for 35 years.

Yeah. And this is why notebook import aspires in the flattened Selkirk's false case. To do something much simpler, which is just use sub lists.

Well, how does it know I don't.

Understand that notebook import. It should be said notebook import is just giving you the grouping data. It's not giving you the state of the groups. Presumably nobody cares if you're dealing with notebook importance.

I'm not sure that's true.

Well, my point being that if you cared to that degree, you would go to notebook read, because notebook read is the high fidelity, no data loss system.

I could argue that notebook import, and I assume Lou is here, that notebook.

Import Lou is not here. But I. But Lou and I have talked. But Lou and I have talked about.

This all Right, but I mean, to me there should be a version of Notebook Import that says get me only the visible cells.

No one has ever asked for that.

I can imagine that being useful because that's the. Get me every cell and get me only the cells that are open, visible.

Those are not the same thing. Visible and open.

Yeah, I get it, I get it. But I mean, that's a finite gradation. But I mean, get me all the cells I can see. And now the question here is, do I want them as the high fidelity notebook read thing or do I want them as nested lists?

Well, and the whole idea of Notebook Import is, I mean, you could in principle have always done the sorts of things you do with Notebook Import, but Notebook import is trying to lower the bar.

I know, I know, I know, I know, I know, I know, I know. The only thing that we're. Okay, it's returning a list of cells. Right? And it's presumably returning when. If you say text.

Right, so, right, so you have two. Two in out groups in this. In this thing.

Yes.

And, and you get a list of two of two lists. Correct, Right, and right, so. So presumably if you know, you.

I'm sorry, what's happening here? Why did you. Oh, it was just slow as hell.

Well, it was calling Notebook evaluate to. And I'm not sure why the notebook evaluates sometimes is a bit persnickety in that regard. I'm not sure if that's what happened.

Okay, so this is what we see here.

Yes.

And if I say I'm totally confused by this, I'm totally confused.

So. Yeah, so now, now that's the default behavior. You could actually remove the option and you would get the same behavior.

Okay. The default behavior is just bring in the cells in the order they appear in the notebook. I don't care how they're grouped.

Correct.

Okay, So I think.

And to be clear, I'm not asking to change any of this for Notebook import, I'm just explaining it because I want to add functionality to similar funct to other functions.

Yeah, I understand, I understand. Which I think is fine. But I, I don't. This name is really. Doesn't work. This name doesn't work. And the fact that it's a string doesn't work.

Clearly. And I would, and I would happily throw myself on, on my sword if I were involved in the base design of this function, but somehow I got cut out of the design.

So I don't think we ever. I think it was viewed as an experimental function, which I don't know why it isn't. Whatever. This. This should have been changed. But anyway, never mind.

I. I agree and I'm willing to make that happen pending this discussion.

Okay. All right, fine.

Okay, so what would be.

Guys, can we please just focus on what we're actually trying to deal with here? Look what we thought. Go ahead.

Well, I. I am trying to understand how this name will map to other functions. And in particular, the name is no good. Exactly. Well, which is why I'm trying to understand how this will affect other functions. So, for example, in Notebook Read, what will it do?

Well, so it. First of all, in Notebook Read, it's defined fault would be different because we can't change Notebook Read Notebook Read's default behavior. But what I'm proposing is that if you set flattened cell groups, or whatever we call it to true, then it wouldn't give you any of that cell group data stuff at all. It would simply give you a list of cell expressions, much like, you know, what we just saw with Notebook import.

Okay. Right. In which case I feel like flattened cell. So if in, in the notebook import case, if. If you evaluated that third input in the first input cell without the notebook, without the flattened cell groups, would you get basically the same thing as Notebook read with a cell of cell group data of blah, blah, blah.

Correct.

Yeah. In which case I feel like flattened cell groups is com. The completely wrong name. Right. Because it's like. Right. It's your. It's not really flattening. It's like removing or group says groups.

Yes, you are correct. It's basically stripping them.

Yeah.

Well, so why, why don't we have, you know, preserve grouping. Preserve cell grouping. Or you know, true or false strip cell grouping.

Well, I don't think we need two.

No, no, no.

One or the other. Right. I mean, drip. You know, we have these things. What is it called? Strip on. What is it called? Strip on. Input.

Strip on input.

Yeah.

Why am I not finding this? In.

In.

In. Why am I not finding that?

It's a pretty low level option. It. It's possible that it's been omitted.

Wait, wait, wait, wait, wait, wait. Look, look, look. What's going on here?

Oh, it's showing up in my completion.

Maybe you have enough like package symbols that are.

This is a prototype build, so I don't know what I think it's just a bug. So it needs to be looked at. Oh, no, it isn't. That's 13 1, 13 3. Okay, somebody's got.

I think it's losing to all the string p Functions.

Right. You felt sensitive. Completely.

Right.

And it gets downraked because it's an option and you're typing an option name.

Why is it not. Oh, you mean it is there?

We just need to get packed past.

The P. Oh, I see.

Because those are all string P things.

Stupid.

No, no, no, no, no, no, no, no, no. It's not stupid at all.

All right, okay, okay, okay, okay.

It's highly intelligent, actually.

And, and it's doing exactly what you wanted, which is to sort of, you know, not insist on matching cases if we defaulted to, you know.

All right, okay, okay, okay, okay. I'm going to stop complaining about that for a second. Okay, so then strip cell. Strip. What do we call cell. What is the grouping option? Strip cell grouping.

Let's see, we have a couple of. Yeah, we have cell grouping, whole cell group opener, cell grouping rules. We're definitely using the phrase cell group or cell grouping in. In all these cases. Strip cell grouping is perfectly reasonable. I. I'd be happy with that.

Okay, then let's go with this. I mean, strip cell grouping then. And then that can work in all these different cases. All right, while we're at it.

So. So it would default to true for notebook import and cells, and it would default to false for notebook read.

Sounds right to me.

Okay.

Okay, what's this one here?

So. This is basically something that allows you to, well, go down to the options, examples. I think that might be the best place to look it up. So it's basically, you know, a way to. It's basically a way to customize. You know, instead of having the same rule apply to each and every cell, have different rules for different cell styles.

And why don't we call it cell style import rules? Do we have an analogy with that? With import somehow?

Not with imports.

Yeah, definitely not with import because. Right. That's. Might be a related context.

Well, I, I'd be okay with cell style import rules.

Okay. And I can, I can write that up as well. I'm happy to do so. Okay, that's my piece.

Thanks. Thanks. Yeah.

You know, I asked. I asked Ian if I was clear enough, and he told me I needed an additional exclamation point to properly convey my feelings. So that's why there are four.

So here's the issue. What is a good mechanism? I've got an expression, and I just want to store it in the cloud so I can retrieve it at some later time. And I just want a UUID for it. I just want to handle. Maybe there's a different mechanism for Doing it. Maybe it's create cloud expression. I don't know. Is that the right thing?

I mean, why is this not like cloud Put?

I mean, I mean it is cloud put. This literally works with cloud put. I was in a previous discussion with you about this and you thought that cloud put didn't work. One argument, cloud put didn't work. But it does.

And now if I say get of that. Yep. Okay. I'm just confuzzled. Okay. Ignore it. This. Okay, just let's make sure that there's a, the cloud export points to cloud Put.

Okay.

Okay, I'll be right.

So there, there are a couple of, you know, wrinkles there. You know, we just, we discussed this at some point and decided it wouldn't be useful, but we didn't record why and I was unable, I'm supposed to.

Go to some other meeting in just a couple of minutes here.

I mean, if you want to punt on this till next time.

But the idea was Nest F2 of X.

Right.

It's like F to the Power 2 comp. You know, compose F twice and apply it to X. Yep.

I mean, I, I understand what the, the proposal is. You know, we decided at some point it wouldn't be useful. Like I said, we didn't record why. You know, one possible issue is that Nescref and Nestry document the 2 arc form to basically be do it one step. So this would be inconsistent.

I don't think that's a huge problem. Although I think the do it one step thing is a bit weird anyway.

Yeah. Right. And you know, if we wanted to extend this to all of the nest things, then, you know. Right. We would only be able to do the most basic of the nest whiles as an operator form. Right. Because they're distinguished by additional arguments.

We need nest while as an operator form. I think the point here is merely the, the, the straightforward thing of, of, you know, functional square of F.

And.

So would it actually evaluate to F.

Compose F.

That's interesting.

Or composition F comma F?

I don't think so. I don't think so. I think it will be just an operator form that just sits there. Symbolically. That's the most useful thing.

Yeah, I mean, I, I Right. If we do nest, we should also do nest lists because, but sure, you know, if, if, if, if we're okay with them being different from the nest tree and we're ignoring nest while. I don't think, I mean, Danny, do you have any comments on or. Ian, I don't see any earthly reason why we really want this because I.

Want it can make very nice, elegant code. That's the reason. That's why I want it just for code. Elegance. I got a rough. All right, see you guys. Bye.