Live CEOing Ep 687: Language Design in Wolfram Language [Command Line Interface]

Okay, hi everyone. We are about to look at a design for a command line interface and Connor is going to explain this to us.

Right. So we've got a number of things to discuss, but this is a little open ended. I'll just say for the benefit of those of us who work here. This is not something for 13.3. So this is not design review per se, and there's no time pressure because there's a lot of different directions this could go in. So we're just kind of trying to scope things out. But there are some specific things that I was planning to talk about, but that's just my agenda.

Give us the outline of what the hexapoint is. Yeah, right. So there's basically where you type Wolfram on a command line, it does something and you want to expand what it can do to be beyond the 1988 input line output line, correct?

Yeah.

So as I mentioned here, I mean there's basically two use cases for improving command line interfaces in Wolfram language. There's the official command line tool, like you were saying, that's $wolfram or $wolfram script rather. And then there's the case of user authored scripts that want to provide a, a non trivial command line interface with kind of the fancy features people have come to expect of, well, fancy for what they are on a command line. So I have functionality that kind of covers both those use cases.

But I mean, this is a third party Wolfram language piece of code that is going to just like. And maybe you can remind me, Chris, for example, in Wolfram script, when you want to launch a notebook from Wolfram Script, how do you do it? If you're running a Wolfram script command and you want to. Is Chris cooler here?

I'm not sure you can launch a notebook, can you?

That was what we had in mind. Maybe we never implemented it.

I think that's the case.

I remember seeing something in the documentation about that, but I don't think it's ever been implemented.

Ah, okay. But the, the point was in that case it launches a notebook and then there's a back and forth interaction through the notebook interface. And in a sense, what I think you were saying, Connor, is that there will be a mechanism where you could launch a command line program and then have a. Are you imagining a back and forth interaction or are you imagining command line arguments?

Command line arguments? So we're talking about something that is. It is built purposely for. To be used as a command line tool.

Okay, so which TypeScript has done. But we don't have a way to. I mean, beyond dollar command line and parsing dollar command line, we don't have a way to let you specify. I mean, what we need is something like an API function. I thought we designed this.

No, there's two things. Hold on, hold on. So the first thing is Wolfram Script is totally focused on running Wolfram language programs. Right. That's all it does. And all of its options are about where to run them and with what options and so on. Right. I think that's different from what this is here, where the point is to be able to do other kind of auxiliary operations like making packlets and stuff like that, running tests, etc. That's not running Wolfram language code, at least not visibly. Right, okay.

But that's also segues into Connor's comment about users doing things. And I thought we had a plan like five years ago, just like we have. Yeah, we had a plan for a function called Command Line Deploy. Chris Cooley, do you remember this?

I remember it certainly.

And the idea would be.

But Steven, that's really not what Connor's got. Connor's not trying to write a launcher for Wolfram language code. That's what Command Line Deploy was, a launcher for Wolfram language programs. He's trying to add capability to our Wolfram executable so that it can take a set of commands to operations. Like Christopher was saying, build a packlet, run some tests. It's not about launching. It's not about new ways to launch executables that are built with Wolfram language. It's really about.

No, I understand, but I'm going to claim that the right way to think about this, given the comment that Connor made that you want user. This should be a user addable thing. Why not think of it as going to run some code? It's going to be Wolfram language code. It might call other programs, it might do other things, but basically it's a mechanism for saying how do you take, how do you configure a thing? I mean, that was what I understood from Connor or. Right, right.

But the question. Yeah, go ahead, Connor.

If I could just interject. I've prepared, prepared an agenda that covers from the low level parts to, I think the high level parts that you're talking about, Steven. I think I'm kind of talking about all of it, perhaps with slightly different language or from a slightly different direction. But I've tried to cover, I think each of those use cases.

Okay. I mean, so one use case is. It's just like we've got separate things right now for dealing with license managers and God knows what else. And you want to wrap those all into a single command line function.

Yeah, basically.

Okay, make it.

I mean, people often are familiar with using the command line for things that are kind of developer oriented tasks or kind of administration oriented tasks. So I think having a. Yeah, I get it.

I mean, it's a vernacular that has changed in 35 years. 35 years ago, you know, Unix was big on having sort of small tools, so to speak. What you're telling me is that for perhaps for reasons forced by signatures and operating systems and things like that, people are preferring, you know, in Unix. Well, I don't even.

Right. I think that change is a whole other thing though. I mean like for example, even apt, like there used to be apt. Get apt, what is it? Apt, Search, apt info, stuff like that. Now there's just apt and you do a sub command to call it, for example. But I think. Well, yeah, yeah, yeah.

Okay. So that's one thing is we need to modernize and package everything. Everything that can be called from the command line. Almost everything you're saying we should package together in this one function. And obviously obvious question is Wolfram Script, should it be packaged into the same function?

I think much of Wolfram Script's functionality should be yes. So the I, you know, the idea of, I don't know, we can just see what's in here. I mean, run a piece of code, execute a file, call an API, run a Wolfram language function. All of that stuff makes perfect sense within this, this, this Wolfram command as well. I think the only aspects of Wolfram Script that are a little incongruous here are, is that Wolfram Script was kind of conceptualized as a thing that can run Wolfram Language code without there having to be a local installation of the Wolfram language. So it's a separate tool. But that's a bit, it's a bit anachronistic and it's a bit.

What do you mean it's anachronistic? I thought it's modernist. I thought it was modernizing.

Well, so the odd thing about Wolfram Script is that kind of has its own way of discovering which installation of the Wolfram language it's going to use if you do have one installed locally. So it's kind of a, it's a second level of configuration on where should this command find the installation of the program it's running, which is a little, you know, I think the more familiar thing is that if I'VE put a program on my path. Like, if I've put Wolfram Script on my path, I would expect it to use the version of the Wolfram language that it was installed with. So, for example, if I were to run, say, Python on my command line, this doesn't refer to, you know, find the newest installation of Python on my system.

I understand, I understand it refers to a specific thing, but look, I think this Wolfram script functionality is really very useful. It's just different and its name is a little bit confusing relative to just a Wolfram binary.

Correct.

I mean, I think it's useful for a more niche case, which is the case of Wolfram. I don't have a kernel installed at all, but I still want to be able to run certain things.

Yes.

Yeah. And to be clear, we're not proposing removing or getting rid of Wolfram Script in any way, just giving us a level of attention to the local command line interface as well. The one doesn't do.

So you're saying many of the commands here would be transferred over to the pure Wolfram function, Some will be left only in Wolfram Script, and maybe the pure Wolfram function would add others. Is that correct? But I mean, things like hashbang Wolfram Script, would that then become. Would there be a hashbang Wolfram thing?

Yeah, I think that would be natural.

Okay, okay. Because I really think there's some very nice stuff in Wolfram Script and I think.

Oh, yeah, I agree. Yeah.

Okay. But almost all of it can be just assumed over.

Yes. Except you're saying the ability to go to a remote kernel. Right.

But the point is that this is like. I think there's no problem here. It's just that we have two different command line executables and they share a bunch of the same operations, but with the difference being that, you know, Wolfram Script doesn't. Isn't tied to a particular kernel.

Yeah, I understand. It's a pretty, pretty. A bit of a weird name for that. And we might need to rename Wolfram Script.

Well, it was all. It was already a weird name.

That doesn't. That doesn't encourage me, but yeah. Okay. All right, so let's. Let's dig into some of what Connor wanted to go over. I think I'm now. I'm now at least understanding where we're at.

Sure. So I'll just go through some things. So there's levels to this. The most basic is I've written some functionality for essentially styling things that you use on the command line. So this happens to just be a package under my name. But obviously if this were made official, this would work a bit differently. But there's a new function called terminal style, which you can give, well, some expression to and specify some styling parameters, and you get output that on a command line looks well styled, as you'd expect. Now, I'm going to mention something quick because people may have picked this out. There's a very intentional reason that this is the string red and not the symbolic specification red, you know, the RBG color object. And that. That is because this is. It's like a pseudo color. This is referring to the color red in my terminals. You know, 16, you know, like table of colors that has 16 colors in it that happens to be RGB, you know, pure red. But it doesn't have to be. It depends on kind of the quote style sheet that I'm using.

Right.

There's a color map. And what I was imagining, though I haven't implemented this, is that doing this uses a function piece of functionality called true color, where it does actually use the exact color you specify, you know, the full 24 bit color gamut.

But. So first question is, why is this not style of blah, comma, whatever.

So I went back and forth on that, though. My concern was that there's a lot of style things that you can't represent on a command line easily. Now, maybe that's not a very good argument, but.

Well, what about size of the font? Can you represent that or not?

No, not something terminals support. I mean, it's very constrained. You've got colors and you've got things like bold, underline.

Oh, so close enough.

Oh, I underlined. Yeah, the errors need to be better, but yeah, and that's basically it. Like colors and basic text styles. Some very fancy terminals support things like images, but not really at the point where I want to rely on that being a feature that can be used. I mean, as you kind of expect. Yeah.

So look, basically what you're saying is if you write a piece of code, it might have styling in it for a notebook, a very small subset of that styling will also just work on a terminal. So one point of view is you just use whatever styling you use in a notebook, and when it's rendered on a terminal, those things which happen to work on a terminal will work on a terminal.

Yeah, yeah, that's one possibility.

I mean, style already does honestly some weird stuff like that, you know, in notebooks where it's like if you give it a style that it knows how to render, it'll Render it and otherwise it'll just be unavaluable, like a font. Right. Whether there's the other direction too, which is the thing of like, oh, I've got a bunch of points I'm feeding to List Plot. I'll use style around them. But if you actually show the points, you know, it's this kind of bizarre. You know, it'll be a coordinate and the coordinate will be rendered in red. Right. As a piece of text.

Oh, God, yes. Right. But yes, that's pretty good. What happened there? Why didn't that come out red?

I'm not sure. I just don't do this often enough. And this is actually. Oh, I wonder if there's a bug.

Oh, well, that bug is more nefarious. I mean, more. Okay, yeah.

Anyway, that is a point, but I don't want to get stuck there.

I definitely favor that style just renders if it happens to be available on the terminal. And part of the reason I say that is because there was a time when terminals couldn't do Unicode. There was a time when they couldn't do this. The motion, the trend in terminals to do more. It's a pretty damn slow trend, but it is a trend.

So maybe one question would be, what if some of the style directives can work and others can't just render the.

Ones that can work and ignore the ones that can't. Which is the same thing that happens if you say use a font that is just unloaded or whatever else just doesn't use it.

But, but I mean, so, Right, let me see. So it would, it would. I see. So you have style of hello comma, red comma, something it can't render. It will show that entire style expression in red. Or it'll show hello in red and it will just ignore the thing it can't render.

No, no, it should just ignore the thing it can't render.

Okay, fine.

Like it's obviously not printing as the out input from the style anyway. So fine, it's red, but not. But not 30 point size.

Yeah, I think it's the right thing to do because I really think we need to, you know, it's only going to get, you know, you said some things do images, there'll be things that do different fonts eventually maybe, you know, and also I can imagine an intermediate thing where you're in a special terminal, you know, of some kind, where people want it to do more.

Yeah, that makes sense.

You know. Okay, all right, let's move on right now.

1.

And again, we don't want to get hung up on this. But the fact like this, this choice of using string red, there is a question of what is the equivalent of that if you're just using style. I mean, it could be something like ANSI red or.

I don't know, I think that maybe.

We could have a symbol that's like. I don't know what to call it. Terminal color of quotes red. Oh, I. I'm not sure that we should have terminal red. Because my guess.

Yeah.

And I don't know, my guess is that there's some color palette. Different terminals will have different entries in the color palette and they'll be named, and they'll mostly have a name red, but some of them will also have a cyan. Right. Yeah.

I think a terminal color wrapper sounds reasonable. Although I'm concerned about what does it do in a notebook?

Well, maybe I think it doesn't. It doesn't really mean anything.

Right.

I mean, it's.

It's. We have default mappings.

Well, but I'm saying again, this depends on something, I don't know, which is if they're very terminal dependent. Right. Like if, If. Oh, I use the, you know, fish terminal or whatever. You know, that's not.

Actually looks like they have standardized meanings.

Yeah, they basically do.

Right. So maybe we get all these standard ones and if you're using some terminal that has some weird colors that aren't on the Wikipedia page or something, then.

Okay, I've got a different point to make. One of the things that's happening is these are defined by escape codes. I might actually want the string. So we need a mechanism for generating that string.

So I think the string with the escape code.

Yeah, I think it should be something like terminal string of whatever will. If you say terminal string of style, blah comma red, it will generate it with the escape codes. Will just return you a string which is the, you know, the terminal string with its escape codes.

Right. It would. I mean, the string itself would have to be escaped though, because. Yeah, I mean, the escape codes themselves, if they appear in the terminal, they're instantly like they're live. You can't kind of inertify them easily.

Oh, I know, I know, I know. That's fine. That's just fine. That's exactly what we want.

Okay. The point is, if you're in a notebook, you want to be able to see the string.

Yeah, but it's just terminal string of that because you might want to put it. You might want to output it, you might want to write a piece of code. That, you know, for example, you might want to save it in a file. You might want to save the thing in a file and then render it in a terminal and see all the brightly bright colors. No, don't serve terminal style. It's just of style.

Yeah, but by the way, I mean, what is the. What is the. Right. More generally, what system does the. Does the command line version currently use to stringify stuff? Because input. No, it doesn't use input form. Right, Output form.

Yeah, default.

So use like two string of expression, output form.

Correct.

So maybe it'd be worth having a terminal form, which is.

Oh, yeah, I do. Said I do. Oh, yeah, that's right.

Okay, so you'd say two string of block comma terminal form. And that would give you the thing that's escaped with all the. Right. Terminal stuff.

Oh, yeah, yeah.

That would be very reasonable.

That might actually already work. So terminal form.

It shouldn't be a. Well, okay. Right. It should just be a terminal form. Should be a symbol like output form. Right, right.

It is. That's. That's what just happened.

Oh, I. I see.

Right. But. But you could also say tostring of blah comma terminal form.

Let's see.

No, no, no, that's not the right thing. That's not the right thing. It's two string comma terminal form.

Yeah. Tostring is a second argument, which is the. The. Know the form to use. So.

But I think the.

This. I feel like the thing is that that weird kernel stuff to work.

The problem is that that is exactly what it's doing. And the escape sequences in that string are getting interpreted by the terminal.

Right.

So.

Oh, good. So. So why don't you do this. No, no, no.

In.

In. In the output there.

So.

Right. That's what.

Beautiful. Yeah, okay.

That's exactly right. I mean, it's a separate question whether the. Whether the notebook front end should interpret the terminal escape codes, which I can see an argument for it doing.

Maybe, though. I mean, I'd worried it'd get kind of weird quickly because it's kind of. It's a new kind of escaping. And it's also the fact that like a notebook, we already have things that are far better for use in a notebook. I think trying to.

This behavior is fine, it seems to me. I mean, this is that you can save it in a file. It's all good. I mean, I don't even know what. Actually, there's something a little bit weird about that. If you say full form, not input form. Say full form at the end of that line.

Yeah, I mean, that's just. That's just our normal weird character encoding stuff at work.

Yes, but it's not obvious that raw escape in input form is correctly rendered. Well, maybe it is. I don't know. That's just what happened there. It's just fine. I mean, if we picked up the character codes of that, they'll be signed on what the escape character code is. But it's fine.

Yeah.

Okay, moving on.

Moving on. Right, so that's basic formatting of stuff. Now, I've implemented, and I have. You can see here, I've implemented terminal form formatting for test results. I've implemented it for. You can do URL read of does not exist.com and format that in terminal form and you get a red formatted failure. And obviously you can imagine Alice would extend other stuff, but that's what I've implemented so far. So that's all of this input here. This was just in Wolfram script. I'm going to drop now into this custom interface I've implemented to show off a couple of other things, but obviously these would be in the official interpreter eventually. First, I'll just mention that this. This terminal has terminal form set as the format type of.

Oh, yep. Okay.

All right.

Well, actually, that should have.

Should have. I thought, well, let's do this. Well, okay, there you can see that messages are getting printed in red, which is another thing. Okay, let's try that one more time. Options dollar output. There we go. Now it's terminal form.

Yep. So reading page width from the term cap.

No, but it could. We talked about that before. It's on my list.

I don't know if there's a meaningful page height.

It's probably possible for some terminals to get the currently visible height of the page. I don't know how. Yeah, same thing. Messages formatted red, failures format in red as well. Other thing I'll mention is that print output is understated. Maybe it's a little too understated, but. And also in is written in bold, whereas out is just normal. So it makes it a little easier to visually parse your separated input and outputs.

Oh, cool. I mean, do we have anybody from Graphic Design? If we're really going to look at those things, we should.

No, no one from graphic design is here.

Okay, well, let's put that aside. Let's. You know, there are probably a few graphic design decisions, like how gray is the gray and things like that. It looks like you have two levels of gray that you have available.

Yeah.

And that's somewhat limited by the. Obviously the named colors that are available commonly terminals.

I know you've got two levels of gray I can see.

Yeah. So that's basic colored output on the command line. If we go one level up, this is where we get to actually parsing command line arguments. So I've got this example here, this greeting WLS script. And if I go ahead and start running this. Let me just walk quickly through what this does. So it loads various packages and then it has a symbolic specification of essentially a parser of a sequence of command line arguments. We were just talking about the naming. This happens to be called Clap because I'm using a Rust library under the hood that implements this subtle command line argument parsing logic that happens to be called Clap Clap command line argument parser.

Which is not good, but that's okay.

That's fine. So this is. This is like. It's a link packlet. It's just linked to this library anyway. So symbolic.

This is incredibly close to what we were imagining for command line deposits. Deploy.

Yeah.

Yeah. So I think it's probably mainline deploy. I'll just say that I think that became standalone applications. So it was, you know, I think the command line deploy project became standalone applications.

I think you're basically right. Yeah.

Maybe one, One difference is that. Right. I mean, my understanding is that these, this is for. This is for the Wolfram executable as opposed to for your own whatever executable. Right.

Well, this is just a library. Like, I mean, this is a WLS script. It's. If it's part of the system, anybody could use this. We might have a gigantic one of these that's the whole. The official Wolfram command. Because these can be nested and built up to be arbitrarily complex. But there's no reason that we wouldn't give those same tools to users who are writing scripts. It's the same mechanism.

And like the thing we want to turn into applications are these Wolfram script script files. You know, one of the things. So I think we can just bring this technology back into standalone applications, you know, and then we can get kind of that. Some of that original command line deploy stuff.

That sounds good to me. So, I mean, in other words, I don't know how you specify a standalone application, but this is part of its. I mean, in a sense, but here.

You don't even need to specify. It's just. You just write it in the source of the application. Right.

Well, how does it, how does it come alive?

Well, here I'LL show you. So this is just a symbolic object. Clap command and clap argument. Do not evaluate. Yeah, just symbolic. It comes alive when you call clap parse. So clap parse takes a list of strings and a parser object.

Okay, so in other words, what you're doing is instead of this being something, you know, you have to have an evaluator, effectively an initial, you know, a boot sequence or something. Or, you know, initial. I don't know what the heck they're called anymore. When, when, when you call main. Right. There's a thing that runs when you call main. Used to be called initial instructions back in the day. Am I making any sense? Something goes. That's just the script or something.

I mean, that's just this, you know, when you run a script, it runs the script and it's going to run it until it hits this part that's sort of waiting, you know, waiting for the command line arguments. And it's going to take all those in and it's going to, you know.

Yeah, I guess I get it. There's also the question of whether, well, whether there's an automated built in command line argument. You're saying for the Wolfram command there will be a built in set of command line argument parsing and actions, but you could have your own thing and then you pick up the dollar command line and do whatever you want with it. Is that right? The thing that's a little funky about this is I'm not really getting this. Okay, in Wolfram script where you have a hash bang Wolfram script, the name of that executable can be anything you want. The name of that thing can be anything you want because. And what you're saying here is that this is a way of taking the command line arguments from that anything you want script and doing things with them. Is that right?

Yeah.

And just to be clear, this is a hashbang wolframscript script at the top here. It's just when it gets into code that I control, I'm parsing the script command line.

I understand, I understand. So I mean, you're saying, let me think about this for a second.

I mean, so it's worth pointing out. And this might be what you're getting at though. Tell me if it's not. So there is a difference. Like Wolfram script has compiled C code for parsing its own command line arguments. And if they're unrecognized or you've passed it a file, then it just runs the file which can do whatever it want, it wants. So for Wolfram script. There is a distinction between Wolfram language code, you know, that's run after all the arguments have been parsed, and the parsing step. This just eliminates that distinction. So in its essence, it would make the Wolfram command kind of this almost extremely thin wrapper. You know, all it does is run a Wolfram language program that parses the command line arguments. It doesn't do any kind of upfront validation before starting a Wolfram kernel. Another way to put that is that right now Wolfram script doesn't have to start a kernel. If you pass it incorrect command line arguments, it can detect that they're syntactically invalid and give you an error without ever having to start a kernel. In this new mechanism, assuming it's specified this symbolic way, by definition, it has to launch a kernel before it can even know if you gave it correctly.

Yeah, I got it, I got it.

I think that's correct because it's, it's, it's right. It's the same as C, right? In C of your main function, it takes argc, argv. It's just that here you access ARGC and ARGV through this dollar SCRIPT command line symbol. Yeah, yeah, yeah.

I mean, one practical consequence is that if you enter incorrect syntax, you have to pay the, you know, 0.7 of a second startup cost of the kernel to see a syntax error, which is not terribly great.

Okay, okay, look, look, look, look, look, look, look. We've got multiple things here, okay?

Yeah.

One thing is we've got a binary, you know, an executable whose name is Wolfram. Okay? That's case one. Case two, we've got an executable whose name is Fish or something, okay? And that is a file. That is a hashbang Wolfram script file. Am I right?

Conceptually, sure. Yeah.

Well, and that hashbang wolframscript file has command line arguments which are not usually. Well, are they usually parsed by Wolfram script or not?

Yeah, I mean, I think if you're using Wolframscript and.

No, no, no. If you've got a hash Wolframscript file.

You'Re saying everything MyScript WLS. If I write MyScript WLS Space Foo Bar. Oh, right, yeah.

Wolfram script does not touch those.

Okay? So that's the case. That seems to me the most interesting for what you have here.

Right? That's exactly what he's doing here.

Okay, so let's just understand what we can do. Okay, so one case is this is a way of processing dollar command line, right? Dollar script command line or whatever.

Mm.

And what's the difference between dollar script command line and dollar command line? What's the difference between those two?

Hold on. It's formatting an output form. Never remember exactly. I think that this, this might drop the command name, but I'll just try it.

Dollar script command line is an empty list for me. But dollar command line is something meaningful. But it'll probably be different if you're calling with Wolfram script.

What are you doing? Just. Just do it in the notebook. It doesn't. You don't need to do.

Oh, well, here, here it's empty because this isn't a script.

So when wolfenscript does dash file, it starts the kernel in a big complicated way. And I think it might be using math lane. So it's dollar command line isn't very helpful for a script. So we set up dollar script command line with the arguments to the executable to the, you know.

Right.

The actual original to the original thing that was typed on the command line.

Right? Yeah. So here you can see the difference. So like Chris was saying, dollar command line is this big long math link thing. Script command line is the actual useful st.

Okay, all right, fine. All right, fine. So what you've got is a utility for parsing $script command line.

Yeah.

Right. Now, in the case of the Wolfram binary, the Wolfram executable, I have no idea what you get for dollar script command line. You appear to get nothing, but presumably.

In that case, what you want to parse is the dollar command line, not the dollar SCRIPT command line.

No, but what you want for dollar script command line is probably those things which are not peeled off as parts of the built in Wolfram function. Peel off the built in stuff, then look at what's left over, then feed that to the. I don't know. Well, it's wrong to me. It just seems wrong to me. It seems like what you should do is if you want to give actual command line arguments. Okay. In the Wolfram binary, there's a bunch of built in stuff, either now built in or stuff you want to add. That's about authentication and packlets and God knows what else. Okay, that's bucket number one. Bucket number two is user added command line arguments, which I argue should be prefaced by a dash, dash, dash or something. And then it follows that with user command line arguments. Am I making sense? So you distinguish them from the other ones, because otherwise, the day you add some random additional built in thing, that blows up a bunch of things that people have done as users it's going to be a very bad day. Do you see what I'm saying?

Right. So we'd have some convention for like you do $wolfram. I mean, bad name, but custom command and then it's some user.

Well, I think it would be. I think what you would do is you would say dash, dash, dash. You would give the commands that were, you know, intrinsic Wolfram binary commands, and then you would end that with a dash, dash, dash. And then everything that follows that would be user commands. That's what I would say.

Well, I think the proposal here is that instead. Well, first I think you do dash, dash is what? Some. Some.

Yeah, okay, fine.

But. But instead of that, you could just frame it as a sub command, which is a lot less weird. Right. So you have a sub command that's basically saying, you know, we'll have Wolfram space, make packlet Wolfram space, you know, run tests, and then we'll go from space, you know, my, you know, custom command. And now you're in sort of custom command land and everything you type after that will get passed as some. You know, instead of it being dollar command line, it will be dollar, I.

Don'T know what you call it. It'll be dollar script command line maybe.

Okay, sure.

Maybe.

But I think the script command line will just be my custom command. It'll be that sub command onward or something.

Yes. Yeah, that's what I would think.

Yeah.

Maybe its first argument will be the whole of the stuff that precedes that as a single string. Just like in the previous case of $script command line, you have the name of the binary followed by the, you know, the arguments that are then possible or manipulable.

Yeah.

Now, for people who are adding custom sub commands, I don't think it would be so weird to just give them this full list because if that's like if they're writing a custom sub command for the Wolfram command, it seems reasonable. And there are fact use cases where some people will have a command that has different behavior depending on whether it's like it's designed to work both as a sub command.

I understand, but I don't think we should. Yes, you can do that. If you say dollar command line, you'll get the whole shebang, the built in commands and the user commands, everything. If you say dollar script command line, you'll only get the user commands. I. It's just really important that we not have something where there's any possibility of confusing a user tag with a built in tag, because otherwise we're boxed in. We can never add a built in tag. Well, I think it has the possibility of blowing up user code.

Yeah. So I think the analogy is kind of like, I mean, we add new context to the system all the time. Right. But we don't worry too much. Like, you know, somebody external, user, theoretically. I know Packlet Tools package. We're happy to collaborate because we think that's pretty unlikely.

Yep.

But same thing here.

I don't think so. I don't think we should assume that here because I think people will use short, you know, dash X, blah, blah blah. We should not assume that.

That's why I'm saying you're talking about something different though. You're talking about like a flag. Right. Flags. People have very, you know, dash X type names. But if you have a whole sub command, I think I would not worry so much about sub command names colliding. I think that the point is the user's code is only triggered if you're using their sub command.

Yeah, I understand, I understand. Well, okay. I would think it was better you can have it as a sub command, but, you know, or you can abbreviate the subcommand with dash dash or something. That is the user sub command thing.

Well, I'm not sure what you mean. I mean, so first, just as a piece of modernity or something, I think flags are generally kind of not really on their way out, but they're not used nearly as much as they used to be.

Okay, well, fine. But then, then what I'm suggesting is what you say is, I know what that Wolfram Dash anything is that's not relevant. Oh, dash cli.

Okay, that's just mine.

Yeah, yeah. Okay, fine. But then, then if it just says Wolfram Dashiell I whatever. And then, you know, custom. If you want to give it a name, just custom. And then after that, everything that follows is part of that sub command.

Right.

Making sense. And then the custom thing, maybe it needs a name that reflects the fact that it's going into dollar script command line.

Wait, to be clear. Also, are you worried about like flags that come before the custom that. That, you know, it's not clear where they go?

Yes, I'm sorry.

I think most of the time you're not allowed to have flags before the sub command.

Really?

Actually not. That is possible with this library. It's actually default. You might do something like that.

Yeah.

Okay, so what I think should happen when you do that is this dollar SCRIPT command line should give you the stuff that's after it should give you the first element of DollarScript. Command line should be the name of the binary, which should be all the way up to the word custom and including the word custom. Okay. That whole thing as a single string.

So something like. In that case, script command line would be like Wolfram CLI b custom.

And then the next argument would be notebook website and the next argument will be build. Right, that's what I'm. Okay. Whereas dollar command line would be the whole thing broken down, including the dash V and the custom as separate elements. That's what I'm suggesting.

Right.

The only suggested change I would make is that this. I think if we want it to be the first element, fine, but I think we should break it down into its constituent components. Because if this is something like pretend.

It'S not that, make it a sub list, then make it a sub.

Yeah, all right. Parsing of these is not trivial. So.

Yeah, all right, but that's sub list there. Okay. Okay. So. So then. All right, so just so we're clear then, if it's a hash, bang, Wolfram script thing, dollar COMMAND line and dollar SCRIPT command line will be the same. Right?

Right.

Wait, wait, you said if it's a.

Wolfram script, if it's a hash thing.

Oh, then they'll be the same.

If it's a Wolfram command, then they will be different.

Right.

So.

Okay, now. Now let's go through.

Actually, sorry, it should be technically, hashbang Wolfram script should still have a different script, command line and command line. It would be, if you do hashbang Wolfram that they'd be the same.

Oh, yeah, okay. Right, fine. Okay. All right, let's go back to your clap library thing.

Well, let me just make sure I got this recorded. Right, so we're saying a custom. I mean, basically script command line, I think you said, equals $$ command line, which makes sense because there wouldn't really be any way for them to be different. Right.

We should move on, I think.

Yeah.

Okay.

Yep.

There may be some further details there. But. But let's go back to the. Okay, so now what you're providing here is basically a function that is very similar to URL parse. Why don't you look up URL parse, please don't look it up in the command line. Just look it up in the. Look it up properly. Just press the. Press the I. Let's actually look it up there so we can see the details and options and so on. Okay, so this makes sense to you. Look down at what it does in.

The Examples, Right, yeah, I'm familiar with this function.

Right, but so is this analogous to what you're talking about?

Analogous? Yeah.

Command line pass.

Right. Let me show you what this actually does.

So, yeah, maybe one difference is I assume that with the command line you have to tell it more about what you're expecting. Right. Like if, if you say, oh, I'm expecting another level of sub command here, or if you're saying, oh, I'm expecting a sequence of strings, then it needs to parse that differently, unlike URL parse, where it kind of parses everything the same every time.

Yes. Yeah, but then how does this relate to the way that, for example, form function is a UI based way of parsing things?

Yeah, I think it's more like form function where you sort of have to. Right. You have to sort of tell it about what you expect. Right.

It has a language. Go ahead.

It's like API function or form Object or even interpreter, where. Or even just a generic pattern like whatever thing you specify in the command line optionally ought to satisfy some spec.

Yeah. Okay, fine, fine, fine. Okay. This looks very hard to understand, your version of this.

Well, that's why we're having a meeting to discuss. But I mean, the basic idea is, I mean, first thing is a name, and I had the usage. I'm gonna. Like, this is the usage message for each of these currently, if you want to peruse that real quick.

Well, I don't understand this at all. I mean. Okay, so we've got, we've got a list and it's got things in that list. Right.

Sorry. I mean, basically that parser, the whole clap command thing is the same as the, the first argument to form function. Right. Or really, I think it's a form object, isn't it? Yeah, I think map command is the same thing as a form object or like, you know, conceptually.

Yeah, it's very similar. It's a symbolic specification of kind of a grammar, I guess, that you need to satisfy with your arguments.

Right. And it's a little stylistically different in that form, form object kind of maybe bends over backward to not use any special symbols. It's all just like lists and associations and stuff. Whereas this has some of its own symbols.

Yeah. Which may or may not be.

Yeah.

Now may not be. Right. Okay, so let's, let's take a look.

At.

Okay, could you scroll up to the top of form object again, please?

Sure.

Okay, so it has things like defaults and auto submitting. Okay. And whether. All right. Okay, so hold on, let's Take a look here. Okay, so I have a question. Why is it the case, you know, in dollar script command line, right now it's an inhomogeneous pile of, you know, argument elements, you know, this, that and the other broken into lists. Don't we already know or do we not know? Do we? Is it the case that the only thing we know is that it's a collection of tokens separated by white space? Is that the only thing we know or can we deduce? Okay, so it's a collection of tokens separated by white space, correct?

Yeah.

Okay, so now what kind of a grammar is allowed in describing the collection of tokens separated by white space?

Right, and so that's. That's right, wrote a little bit on this. But I mean that those conventions have changed over time, right? So like, what we see in Wolfram and Wolf script is kind of the. The single leading dash specifies, you know, a top level switch between different areas of functionality. That's kind of fallen out of favor for a sub command name that doesn't have a leading dash. So, you know, this is arbitrary in some sense. But the grammar that's common today is you have a sequence of command names and sub commands that don't have any prefix, you know, so Wolfram Packlet build is like a hierarchy of sub commands that nest together and then those can have two types of arguments, positional arguments and non positional arguments. Positional arguments always have to take a value. Non positional arguments don't have to take a value.

How are they specified? With flags with bash.

Non positional. Right. Non positional arguments always start with a single dash or a double dash.

So it's a key value thing. So you say dash foo and then space bar, and that's value key foo, value bar, Correct?

Yeah, So I don't know.

Okay, so that's a mechanism we don't readily have in the language is something which is an alternate of named and unnamed. But essentially what you're trying to do is you're trying to fill certain arguments. You know, can we think of this thing as a function where you're trying to fill certain arguments? I think, yes.

Yeah.

Okay, so the naive thing that you would do. Well, I don't know. Okay. I don't know. I have no idea what the scrammer should look like. What on earth is this thing that you have on the. Right, just what on earth is it?

Sorry, this thing or.

Yes, what on earth is it?

Well, why don't I describe it and we'll see if we get so the basic idea and I'll pull up the usage message so people can reference this. So a command is made up of. I mean, there's essentially, if you don't define a command, you have to know its name and you have to know its arguments. And optionally it can also have sub commands. So those are the basic elements of a command. And these sub commands are themselves, at least in this representation, clap command instances. So there's this arbitrary hierarchy of commands with arguments and optional sub commands. Arguments have a name and they have some action that represents how they should be processed when the parser encounters them. So this. Right, so that action. Okay, there's a couple of options. It can either, like this is a named key. It can either set that named key to some scalar value.

It's just freaking horrifying. It's just horrifying. Look, the thing should return basically some association, just like form function does. It shouldn't have any of these kind of weird shortcut. Oh, I'm going to execute at the time I'm parsing type things. I'm making sense. The goal of this function should be to return an association that represents the thing that's supposed to be represented by all those command line arguments.

Right. There's some nuances there though, because, for example, if you specify this up front, the computation essentially is in the symbolic specification. You get things like error reporting for free. Right? If you try to specify multiple values for one of these things, Clap parse can handle that. If you've told it whether you want to allow multiple values or not, you.

Can do that in the case it's the exact same thing with form object. Exact same thing. Oh yeah, execute in the middle of this thing. It's a big mistake. You want to separately say, okay, that's there's two steps. This, it's like natural command line understanding. Okay? Just like we have natural language understanding, we don't try to. We don't have an evaluator that runs at the time of the of parsing. You know, it's like you parse, first you make an expression, then you evaluate it.

Right? I guess maybe I'm not understanding which aspect of this you think is evaluating and violating that principle.

And the thing about setting, that's just crazy. You just want to have something where the goal of the thing is to take, you know, everything has a key and a value. What you're going to return here, I assume is, I mean, anything, which is actually evaluating anything at the time of parsing is a mistake, right?

This is not set as in system set. This is terminology.

But it's silly because it should just say look. What this should look like is. Okay, the problem is, okay, let's just understand what a possible grammar looks like. So you've got something which has commands and sub commands and for some reason a sub, sub comm.

So I think I'm trying to think, is it restricted? I mean, because I think, I don't want to get hung up here because this, this is not like this is kind of equivalent to a different specification that might be something like this option takes a scalar. Right? The fact that it's called set and that sounds mutable and scary I think is that's just happens what it happens to be called in the clap library. So I mirrored that. But conceptually that's not relevant. Yeah, right, it's not relevant. I think what I want to focus on here is semantics, not the name.

So what would you call might be.

I mean, I don't know exactly. It might be something like scalar or required value or it might be the default. And if you want anything other than that, you have to specify something like.

Just be clear of what we're trying to return here with commands and sub commands and so on. Is it or is it not the case that what we are trying to return is a flat association of keys and values? It's. Is that true or is that not true?

I would say not true. So let me show what this actually returns. So if you do clap parse, what I have it returning is a list of essentially pairs between command name or sub command name and an associations. This should actually just be association queue though. I don't, I don't need it for this.

Show it. Just show us a return value, a return value from something with sub commands.

Okay, well here, how about I just echo this then I'll run this script. So let's say I pass it, say hello, why don't I copy out this extra stuff?

By the way, this should just run in a notebook. This should just be a thing where you give it a string and it returns whatever expression this is, right? Is that not the case? So you're not messing around with prints inside inside scripts, right? Can't you just take this function clap parse and just run it and it returns an expression?

You can, yeah.

So let's just do that. So we're not messing around inside this crazy harness here. Just take it and put it in a notebook and let's just give it a string that is your dot greeting Hello. Just take that out, put it in a notebook. I don't understand this at all.

All right, well, here. So, okay, first argument is obviously greeting. We're going to call the say hello sub command. Load this. That's just inert.

Yep.

Right. So that's what you get.

Okay, let's understand this. Okay, let's not look at the thing on the other side. Let's just look at this. Okay. Okay. So what is this saying? I don't. So I don't get it at all.

Let me explain.

And I think it's a mistake.

It's similar to. Well, I mean, I think it's. It's just a. It's.

Right.

I mean, it's just another structure, like what form function does or like what form page does.

Form object, but it's messier. Okay.

Form object, for one thing. As far as I know, it doesn't have a notion of hierarchy.

No, it doesn't.

No, it just does.

Okay, so that's one reason it's more complicated because we have to represent that this is a thing nested inside of.

I got it, I got it. But so what's that name and language? Oh, that. That's just your setting there.

Right. Those are. Those are kind of the conceptual, like the name of this argument. So this is the key under which you will access it in the association, in the parsed result.

All right, let's see what this looks like. If there's a sub command.

Well, there is a sub command that's. That's say hello. If I don't pass it a sub command.

No, no, no, no, no, no, no, no, no, no, no. You said it's hierarchical. Show us another level of the hierarchy.

Oh, sure. Okay, so if I wanted to add another layer of sub command, like maybe it's. I don't know, say hello. Like the command would be something like. Okay, I'm gonna add another layer of hierarchy. So say hello has.

Why is that another level of hierarchy that just looks like it's the same level?

Well, it's nested inside this say hello command. Right, So I see. Because again, the.

No, I got it, I got it. So to just.

This is kind of a mess, but.

Put in a command there so we.

Can see what it does. Say hello.

Bye, bye, whatever you want to do.

Bye. Sure. All right, so that's specification for that. I just need to say it takes zero arguments and now.

Put it in say hello and then buy.

Right. Bye. So now we end up with an element three element list, whereby that's not making Clear.

That buy is a sub. I mean, you could imagine a say hello, say, you know, bonjour. And each one has a by sub command, but that's not represented in that output expression.

Right? This is like. So this is kind of like a file tree, like a file system tree, right? It's the full everything on your file system. This is like a file system path. It's a specific sequence, you know, a specific set of things.

Did you understand what I said? If you've got both say hello and say bonjour, and they both have a by sub command, that representation does not indicate. That only allows, you know, it doesn't indicate which command that sub command is associated with.

Right?

I'm saying.

Yeah, and. But I think that's. So let's look at this in a different way. So. So greeting has two sub commands, say hello and say goodbye. This sub command, I'm just call it fucs. I don't know what it would be. But these are. These are at the same level in the hierarchy. So they're elements in this list.

I know, I know they are. But. But the foo is not. Make the second thing fu. Not buy, not by. Make the second one. And then say goodbye. Make its command foo as well.

Okay.

Okay, now show us what that returns. Okay, now. Now put. Now say say goodbye in addition to. No, no, no, no, no, no, no. Not, not both say hello fu and say goodbye.

That's what I did. This is say hello.

No, no, no. In a single line. In a single input line. Say goodbye foo, comma, say hello, fu.

Sorry. Like. Like that.

Two different sub commands.

Yes.

This is not grammatically valid. You couldn't write this.

Why?

Because you can't execute multiple sub commands in one invocation. It's like saying, write a single, you know, symbol that calls two functions at the same time. You can't. It just doesn't.

Okay, so there's only sub commands work. Okay, so that's only. There's only one command which then has sub commands.

Yes.

So in other words, in that parsing thing at the top, those are alternatives.

Yes.

Say hello and say goodbye are alternatives.

Yeah.

Okay, how does grammar rules work? Does it not have a mechanism for this and does it not use the alternative symbol for this?

Well, it does have the alternative symbol there, but if you look at any examples of grammar rules, you give it usually a big list of different rules things, don't you?

Yeah, I think you do.

Yeah.

Right. So this is basically saying the appliance command, you know, matches. You know, matches this. This grammar and the grammar contains.

Okay, so. But this is a very similar setup, right? Turn off the stove or something. There is like commands and sub commands at some level that makes sense.

Yeah, I'm just trying to.

All right, so what you're telling me is that fundamentally, I mean, even though with flags and things like that, there are multiple different things that can exist on a given. There's only one command per. Are you telling me that the way this parser works, there's only one command per command line invocation. Is that correct?

One invocation? Yeah. One command.

There could be multiple flags. Yes, but the other arguments we're dealing with, there could be multiple things of that kind.

Yeah.

Okay, so look, I've got another analogy. Okay. If you look at. Oh, my God, what's it called? The thing that executes shells, the shell from. What is it called?

Run process.

No. Run. Run process. Yeah. Run process.

Yeah.

Okay, so see what that's doing with. With exit code. Now run process. Okay, so run process is just giving the white space delimited elements of the command line.

Yeah.

So what you're saying is you're at the other side of that, parsing the command line. And let's be clear, there are two types of things in the command line. There are flags that occur in any order, and then there is your command with its sub commands. Is that correct?

Yeah, and also positional arguments. But yeah.

What kind of positional arguments? Top level. Positional arguments or positional arguments within commands.

Positional arguments within commands. I mean, everything is within sub command, right? There's like. There's the root command, then there are sub commands. So every.

There's also flags which are not in the. In the sub command.

Well, flags are a type of argument to a command, right. They're kind of like our.

I can say dash V and then say say goodbye.

Yes, but dash V is an argument to the greeting command. So that's. What's a little weird here, is that you can. You can specify.

I go what you do. I get what you're doing. Okay. All right, so.

Which is why you end up with this nested structure where each command has associated argument.

I get it, I get it, I get it. Is it obvious we shouldn't just use a tree?

I think it's all too weird.

It's too weird, probably. Look, I think that the more common thing we would do is these days is to use rules rather than nested lists. But I mean, what you're saying is. Okay.

Right. It is basically a stack, Right?

What you're saying Is this is a one sided, a very one sided tree.

Well, this output, I don't, I don't know that I would call it a tree.

Yeah, I don't think it's a tree. I mean it's just, it's just a, I mean it's literally, it's quite literally a stack. You have a command and a sub command.

Okay, well here's the, here's a way.

To do it, right, and arguments at each level.

Okay, here's the way to do it, right, here's the way to do it. Okay, the question is, can a sub, sub sub command repeat the name of a command?

No, I mean, the name could be the same, but it wouldn't have to call the same code, if that's what you mean.

No, no, no. But could there be two foos there? Because otherwise the structure that I would suggest is something that just is an association.

This would be weird.

The way it works. These sub commands are like, they're kind of like contexts or something, right? It's like saying, you know, okay, we're going to run. This isn't a very good design, but it'll be something like this, you know, Wolfram space, packlets space, create space, some arguments and then separately you have wolfram space, packlet space, you know, build or something like that. So packlet is a sub command and then create or build, whatever.

I guess so scanning until it gets to that next subcommand. So, so one way to represent this type, Type this in. Okay, Is, you know, association. Okay, Greeting arrow, null, association comma, you know, fu arrow.

No, but there can be, they can be the same one.

I know there can be collision.

But, but then in addition, this is also just conceptually wrong because, you know, again, it's like there really is an ordering that's important.

But you get the ordering here. This is the ordering. There is only one. It's a, it's a tree that has to go down just one branch. It goes sub command, command, sub command, sub, sub command, sub, sub, sub command.

The thing is, the thing is, realistically what you're going to want to do is you're going to want to say, you know, for example, maybe with pattern matching or maybe with whatever else you're going to want to say, oh, if the command is, you know, quotes packlet comma quotes create, then I want you to run this code. And if it's quotes, packlet comma quotes build, I want you to run this other code. This is not a useful structure. Doing that kind of.

Okay, okay, okay. So then, then.

Which is what's happening here? Just to illustrate, you know, I'm. I'm doing.

I see it, I see that. Yeah, I see that. But this is. I don't mix the execution with the pausing of the thing.

I don't think it's mixing here. Right.

I don't think I am.

Okay, we got a separate function that just does. Ok, fine. You got that separate function topparse does.

All the parsing and it returns this basically stack like representation and then replace is just actually running something based on that representation.

Okay, all right, so the question is what the most convenient representation is given that what you are typically going to want to do is treat. I mean. Okay, so here's a, here's another radical idea. Okay, what about you go back to the left hand side there. Okay, what about something which says. I know we can call it script command. Okay. Function, script command, of greeting, of quotes, greeting and then no, no, no, no comma there. Now it's a sub value. So now of script command.

This is very XML element.

Well, I'm just trying to understand how you use it. Right, because.

Yeah, I mean, this seems like a worthwhile exploration. I don't know that I love this syntax, but.

And after greeting though, you would have the flags or something, right?

You have like the script. I mean, this might be overly verbose, it might just be.

No, you don't want script talk, you just want to. You just want an association there.

Right? Yeah.

Or it might have to be a list because maybe you can have flags.

More than once, which is horrifyingly xml. Like.

Yes. And also again, the list is I think semantically more appropriate because these things can be. They're positional. Right. The order is important. I know associations are ordered, but it's implicit.

Also, I don't even know how it works because, you know, not all flags have values. I mean, some flags are like, you know, you'll write, you know, dash n space 12 and other ones it's, you know, just dash V. Yeah. Or maybe, Sorry, maybe you're already explaining that in the part. In the.

Yeah, okay, but I want to know how do you typically use this? So this is, this is an alternative.

So, you know, it might be useful, Connor, like if you could just show like Cargo's sub commands just to see like roughly what these. How these hierarchies work.

Sure. Well, I mean, if I. So Cargo H. And for people to know, Cargo is the Rust language build tool. So Cargo's got cargo build.

Or just, just look for a second Though that, so that the first are the flags, then they're the sub commands. Right. And so now try, try doing one, you know, cargo test or something like that. And let's see, it's help or.

Oh, it's help. Yeah, yeah.

Right.

So cargo test, that doesn't have any sub.

Sub commands, but it has lots of flags.

Yeah.

So where do you put those flags? Where do you put those, those flags?

I would say cargo test, maybe dash dash, you know, features something or other.

Why isn't this.

That'S not.

Well, the, the convention. Okay, so this is a weird. It could be new thing.

It could be. Let me see.

So, right, so single dash. This is actually something that Wolfram script is kind of annoying about that. It, I think it does not follow the modern convention. So basically in most of the GNU tools you can do something like, for example, you can do LS lah or something like that. And that's the same as doing L dash a H. Yes. Right. So what that means is you cannot have in a command like that a multi letter flag. And so the convention is normally that single dashes are for single letter commands and if you do dash lah, it'll expand that. And double dashes are used for multi level or multi letter.

Fine.

Yeah. And what's happening here also, like what's represented? This is kind of the convention for this. This is something that has a quote, short name and a long name. So I can say dash, dash, features.

I got it, I got it.

Yeah, but maybe, maybe is there another one like cargo build or something that'll have a sub. Sub command? I don't know one.

I just want to know how you're going to use this. In other words. Ok, you've got. Basically what's going to happen is there's something that's specifying a command here and that has got to dispatch to some actual piece of code. The replace thing is not the right way to do that.

Right. I mean, I, I had thought of something like a command pattern, something or other. And there might be, you know, clap parse and again the name suck. But clap parse, replace and you've got. Actually I might even have an example of. Did I save.

No, no, no, no. Look, look, look, look, look, look, look, look, look.

Well, something where the replace action and the parse action is one function that's made to make it more ergonomic.

Okay, well, hold on.

No, that's the opposite. Just, just, just earlier you were saying that we should not, you know, that the parsing and the execution should be separated. Right? Now you're saying the opposite.

No, look, look, no, I'm saying once you've got the parse done. Ok, here's another case that we have. Another pattern is event handler.

Which also is only very simple events.

Yes, I know, but the main point is that then there's a list that says event name, colon greater than action. The best thing here would be if the executor of this is basically a great big list of. If you see this thing in the command line with certain pattern variables, what one would like is something where you know, What the hell is this?

I'm just thinking out loud. This is. But I mean, all the syntax here is wrong. But you could imagine something like a specification like what you were saying, where the handler is specified in line on the right hand side.

Okay, guess what? That's like. That's just like API function.

Yeah.

Where it's picking up certain arguments and then using them.

There's an important difference though, which is that API function, ultimately there's one function you're ever going to call, sometimes maybe you'll have it so that depending on an argument, you pass the API function, it'll dispatch over many different functions. But there's no clean way to do that. You write a big switch or something like that.

Well, right, URL dispatch has that URL dispatch. Dispatches on. With rules, dispatches on. On a. On a sub, you know, element of the URL.

Right, yeah, right, but that's that you give it different API functions or form functions for each. You know, you don't put the URL dispatch inside your API function, put it outside.

Right, okay, look, look, here's an important point here. This interface, this command line interface is not very combinatorial, unlike what we're used to in language design, which is deeply combinatorial. This is something where there's a command, a sub command. This is a very. As you said, it's a stack or if we draw out all the possibilities, what do they look like? They look like a tree, presumably.

You.

Know, in other words, what are all the possible commands, sub commands, etc. That you could get? Right. At the top level of the command level, there are five different commands, let's say. Or I don't know, the initial command might be one, one thing, but then there are five sub commands. Each of those sub commands has variable numbers, 2, 3, 5, whatever it is. Right.

And yes, it's a tree and basically it's. If you're. It would be super weird to have it turn into a Dag. And if you did have it as a dag, I think you should just have to copy your code like it's.

Yeah, okay, fine. So it's a tree. But now the question is, when you execute that tree, what you want to do is you go to a leaf of the tree, and each leaf is a different execution function. There's no sharing of the execution functions. Each leaf which has picked up certain command, sub, command with various flags goes to a leaf. The leaf knows it has these flags, and it was this named leaf and then has some code. And so to me, what. What needs to happen is each of the flags picked up at each level, you know, what do you feed to the leaf? I think what you feed to the leaf is an association. That can tell. Maybe it's a list, I'm not sure. Or list of associations. Maybe it's a list of associations from each of the levels. And. Okay, look, here's what I would propose, okay? That the full thing. No, no, look, look, look, look, look, look, look, look. You got to decide. The flags are not going to determine what. Okay, let's take a.

Right. The flags shouldn't determine which leaf you end up in.

Right?

Agree with that.

So. So, okay, so every leaf is a sequence of commands and sub commands. Okay? It's labeled by a sequence of commands and sub commands. Do we agree? Yeah, okay.

I think so. Yeah.

So then what I'm suggesting is that the execution layer, like an event handler, is just a great big list of basically commands and sub commands. Okay? So write it out, write it out. So, list of, and then a sub list here. Actually, you can just have an association here. It isn't a list, it's an association. The whole thing in association. No list, no list, just an association. Okay? The first element is the list. List. The key is the list. Top level command, whatever it is. Command comma, hello, comma, you know, sub command one, comma, sub command two. Yep. Arrow. And then. Right. You're being given at that point. Yes. Flag one, blah, blah, blah. Now, now those. That might have to be a set of sub associations because those flags might have repeating names. So you might say command 1, arrow, flag 1, blah, etc, etc, Etc. Right, right. No, no. So to write that out in that association, that association itself. No, no, you don't want. No, no, no, no, no. In that association is. It's an association of associations, okay? The first element of the association. No, there's no flag. It's command one, arrow. The first element of the association is command one, cmd one, arrow and then the flags associated with CMD one.

Right.

Okay, and then see, sub command 1 and the flags associated with that.

But then you have to Repeat, you know, Command 1, Sub Command 1, every, every sub command of Sub Command 1, right?

What do you mean?

Well, Command 1 appears in here more times than necessary because like this keys of this association.

I understand that, I understand that. And possibly then, then one could elide that and just make that into a list. Right? You see, so I'm saying rather than repeating the keys, command 1, etc, etc, etc. Although I'm suspicious about that, but okay, so that's one possibility. The second possibility is command one, blah blah blah blah blah on the left hand side. And then the right hand side is a list of associations. Do you see what I'm saying? So you don't have to repeat the keys. So get rid of those keys and just make that a list and get rid of the association at the beginning there. Okay, all right, and then it's.

Why not thread it? Or something? But yeah.

Yeah, why not thread it? I agree. No, actually I don't agree. I don't agree. Because those flag values. Oh no, maybe I do agree. Here's what I'm suggesting then. Okay, I don't know whether this works pattern matching wise. Okay, so, so thread them there.

Sorry, I don't. When you say thread, I know the.

Function, but I can't.

Command one just make command one a sub list. No, actually command one. I'm sorry, I'm sorry, I'm sorry,. I'm sorry. Command one, arrow and then flag one, blah blah blah. Actually, then this is no longer an association. This is a different, this different animal.

Yeah.

But that's possible as well. It's just a list. Get rid of the outer association. Get rid of the outer association.

In which case then the way that you look up your particular command or sub command or something would be with pattern matching.

Yeah, right.

And now we're back to something that is essentially what I know, just with rule instead of list tuple.

Okay, but now let's think about how you use this. Okay? What does the pattern matching look like? I don't particularly like option C, but anyway, what does the pattern matching look like to say, you know, what we want is something where you can have a function F, where you very cleanly say F of list Command 1, Command, blah blah blah blah blah. And then you simply pick up. There's no script command, there's no script command. It's just. Look what you're doing is this is a part. There's one command that parses the thing. Okay. And then you're just applying standard functions to evaluate it.

Yeah. Well, I think we should try to clarify right now. There's. There's kind of two different routes we're talking about. There's something like the URL dispatcher route where the patterns are mixed in with the computation specification. Right? And then there's the. There's like the clap command route where nothing about how when you match where, like, where is the computation injected? You know, there's nothing in this.

Listen, listen, listen, listen. We're getting confused here.

There's my. Trying to clarify.

Yeah, right. Okay. When you do a parse, you're going to get a single command, one sub command, one sub command to list out. Right? There's no association of these things. A single list.

Yeah.

Okay.

This, this specification makes sense when you've already parsed something like this Makes no sense.

That's the specific. That's the specification of what to do.

Yeah.

Right. So what I'm suggesting is the function that specifies what to do. The big switch, the dispatcher. Right. Then I'm thinking, how do you write that code? And the answer is one possibility. Okay, Maybe there is some function that wraps around to do this, but the One possibility is command 1, sub command 1, sub command 2, colon greater than. Make a separate thing. Make a separate. There's no parser here. It's just the computation here. Just the handler. This is the handler. Okay, so then that colon greater than. And now that function is going to be passed something with a bunch of flags in it. And I don't know exactly how that works. That. That argument should probably be. Yeah, I think it should probably be. Yes. What it should get is a list. Exactly what we had before. The fact that those are repeated is no problem. In other words. Okay, no, actually, actually, no, I'm. I'm thinking. Okay, so here's what I'm suggesting. There's no match queue. There's no match queue.

Stephen, I'm not using this. I, I'm not using this to represent, like, this argument has to have some conceptual value. I'm just writing this to illustrate what is the structure of the argument.

Okay, okay, okay, okay, okay. But, but, but then. Okay, what I'm saying, that function is passed. If you write below what that function has passed. Underneath there is the association. Okay. I think you're just going to want to write it in an input line. Okay. Association. And then a key. And then the key is quotes. Quotes. No, no, no, no, no. I'm sorry. List cmd 1, comma, flag value, flag 1, arrow. That whole thing. No, that whole thing. The whole thing. The whole list is a key. The whole list is a key. No, no, no, no, no, no. Flag one, arrow value one. Okay. Probably not a string, but it doesn't matter. Doesn't matter. Okay. Okay. Now let's say, I don't know, sub command 3 or whatever. Another sub list. Another sub list in here. Another sub list.

Okay.

Sub command three, flag seven. You know, whatever it is, flag seven goes to. Whatever.

What is the. Oh, the. I think the flag 7 is confused, but is sub command 3. Is sub command of.

Yes, it's one of the. What I'm suggesting is that the evaluator here, for every sequence of commands and sub commands, you're defining a function that is going to handle that. What does that function get given? It gets.

These are all the flag values for each of the different sub commands at all the different levels. Yeah.

Yes, that's right. Because then it can choose, oh, I want the flag from the command level, this flag name, I want to use that. Or it says I want.

It also needs to presumably be passed. Maybe this is a separate argument, but it needs to be passed. You know, the. Just positional arguments to the sub command. Ah, right. I assume it can have. Yeah, well, okay, like create and then the name of the packlet. Right.

It might have that as a numbered thing, perhaps. Or it might have.

Well, no, no, but why not this, you know, you have this handler object. You have, you know, this function, which. Which it gets given the flag spec and it gets. Probably gets given the argument spec and the flag spec, and the argument spec is just a list of all the positional arguments of strings or maybe just one big string, I don't know. And then it gets the flag spec.

Say that again. So you're suggesting two arguments. One that is of positional arguments and the other that is of named arguments.

Yeah, basically.

Right.

Yeah. So the first argument would be.

I would argue the first argument should be the positional ones, but it doesn't really matter.

Yes, I agree with you. I.

Flags are like options, right?

So then the first slot will be a list.

Okay, so how does parsing of positional arguments work? Like, are we. Can we assume that we can, you know, for example, break it at spaces and stuff like that? Or. Or is it just a string and we can't touch it? This is kind of a question about how clap works.

No, I mean like, if you. If you want to. If you have a string with spaces in it. That end the command line. You need to use quotes to group them.

Okay. Other bind.

We shouldn't do any processing like that.

Right. Clap will clap sort of has to do that processing. Right. Like it will break it up into.

Well, it's not even clap.

It's the.

It's the operating system's parsing of the command line because you get past the shell. Yeah, rather. Yeah. The shell handles it. Clap doesn't.

Ok, but this positional argument thing, can't there be multiple positional arguments for command one?

Right.

So positional argument one. The value of this might be some scalar or it might itself be a list.

No, I think what it should be is that the first slot should be an association CMD one with.

No, no, I. I think you can't have multiple positional arguments for. You know, if you're using the sub command, you know, if you're using Wolfram space, packlet space create. You can't have any other positional arguments to packlet. Right.

Okay, I don't completely understand what's possible there, but it's either just a homogeneous list of positional arguments or it's cmd one arrow. It's positional arguments, sub cmd, you know, sub command one arrow list positional arguments. I'm not sure which of those is correct.

I would say though, like non root like commands. Like if you have wool from packlet build. Let's say we shouldn't assume that Wolfram doesn't have a positional argument because it could.

It can.

Probably not common. It's not common, but it could.

Okay, well so then in that case, the first argument has to be cmd1 arrow and then list subsea1 and then a list of the args. Right.

Well, maybe it actually should be list cmd1arrow its args and then list cmd1 comma sub command1 args and so on. That way we consistently. So you'd have it. So that. Right cause the. Right. So you'd have. You can't do it quite like that because if you had two sub commands at different levels with the same name, there'd be a collision.

Right.

So you have to refer to sub commands by the path.

Oh, I see.

What you should just write.

Yeah.

So key is. Key is a path.

Yeah. And then we just consistently refer to sub commands by their path.

Right.

When my case, that's true of the slot 2 thing as well.

But we are. Oh yeah, that's true.

They're just flame together. It could be that hypothetically.

But yes, no, but I Think it has to be right, because it's the same problem of there could be a collision of two sub commands with different levels at the same name. Like if I had Wolfram space packlet space Wolfram. Right. That's a sub command of packlet, then there'd be a key collision. But so sorry, you'd write command one, you wouldn't write just sub command two.

Yeah, right, you'd write command one, command two.

Yeah, command one comma, sub command one comma sub command two. Right, the whole path. Yeah, yeah, there.

So hold on, hold on, hold on. I think the better thing for that second slot would be that there's a consistent top level backbone for the association and it either is the positional arguments or it is an association of the flag arguments. So for the second slot write out the same thing that you had for the up above. The second version of what? No, no, no, for the first slot, but you had first slot, the second version of that, that thing there. Make a copy of that, make a copy of that. And now instead of POS1ARG2, whatever it is, just make that an association that says flag one, arrow blah, flag two arrow blah, et cetera, et cetera, et cetera.

Right.

I think that makes sense. And then, yeah, now you might need to pattern match. It's a little bit ugly to pattern match on those flags, but probably not too bad. Well, ok, this is assuming you're not pattern matching on flags that instead you're just passing those. Any pattern matching on flags is done in the, in the processing of the evaluation function. And the evaluation function is called based on the command sub command sequence.

Right.

I would assume. Okay, so two things here, right. I think first thing is that once we're using associations like this, pattern matching is kind of unpleasant, right? I would assume that, that you would be, you know, you're calling lookup and so on and I think that's fine. It might be worth actually having a different first argument that's instead just the sequence of commands and sub commands. So that basically the first argument just tells you what function am I running here, right. Which will let you sort of dispatch. And then once you know what function you're running, you can then just look up into these associations and say, oh, by the way, what was the value of that flag? Just like you do with like option value and stuff like that.

It might be better than to package the whole thing in association and pass just a single thing to the, to that function on the right hand side. Well, actually, you know what, the command.

So you'd have a different function.

Hold on, hold on. Look at the, look at that form of the handler object. You know what the left hand side is because it was the thing. You know what the function, what the command and sub commands were because they were the left hand side that got you to that function.

Yeah.

So if you really wanted to pass those in, you would say X colon and that thing and then pass that in as pattern matching.

Oh sure, sure. Okay, that's true, that's true. Yeah, fine. So just as long as you know what keys you, you expect to be in these associations, I think it's fine. Right.

But I do want to say that one downside of this approach, and it's not a bad one, but if you've got all of these things on the right hand side, like this is. I think we should discuss one level out of the syntax right now. Like this is kind of. Okay, so this is the handler object that gets past. Ok, so this is, I mean, what would this actually, how would this be used? Would it be something like, you know, so you say, you know, and let's say script, command, line, parse of, you know, args and then it's like this thing is the handler object. And as we just discussed this, there's like.

Right.

This gets passed two things which are. So first slot, it's args and flags.

Right, right.

Well, pause args and I'll say options because flags implies boolean. But anyway, so then you do like what replace on.

No, no, no, no. I think you just do something like pause args of, you know, list, you know, command one. Yeah, sure would look up. But, but with, for pause args, it's, it's safe to just, you know, you know, it's always going to exist. But just try, you know, command one comma command or you know, comma sub command one comma sub command two. Right. Like that's a key though. It's not, it's not part. Right?

Yeah, it's not part. A single bracket.

Yeah, yeah, Single bracket on a list.

Yeah. With a list there of Command 1, Command 2.

So we'd say, you know, oh, that's the name of the packlet we want to create. So then we're going to say, you know, packlet create of that.

And then we want to know there's no value equals. It's just packlet created that thing. There's no side effect.

Yeah, sure. So you know, create. Yeah, create packlit. And let's say there's some flag that's like, oh, do you need to, you know, I don't know. Is this a. Is this a fancy packlet? Right. Then. Then you just say comma, you know, fancy packlet, you know, inside the create packlet. Suppose there's an option to create packlet called fancy packlet. You'd say comma. I'm not sure why you say one.

Oh, no, you're right, because that is. It is one. Right. First positional argument is the packlet directory, and the second is the build direct or the output directory. This is the packlet name.

But double brackets. Right. Double bracket parts.

Yes, double brackets.

Yeah, yeah, Yeah.

Fine.

Yeah. Right, and then.

And then. Right.

And then let's suppose we have the flag that's the fancy. Is it a fancy packlet or not? So just write, you know, comma, you.

Know, fancy arrow and then the.

Okay, sorry. So second handler.

No, no, same. Same handler.

Same handler.

We're just doing a flag. Right. So. So just suppose there's an option to create packlet that's called fancy packlet.

Well, there's extensions.

Well, but I want to do like a boolean flag or something, just because it's easier.

But food resource.

Okay. It's a more realistic one.

Yeah, sure. But anyway, Right. That's a boolean flag. So options.

Now, options of.

And.

It'll be a list, will be two, and then another layer of list, and then the inner list. You're going to say, you know, command. You're going to say, you know, wolfram pack with create.

Because I assume this is going to.

Be an option for create. Right.

Does nobody feel that this is horribly verbose?

Well, maybe it is. If you can have flags at every level, and you can have positional arguments at every level, too, then I think there's no getting around it. Right.

Well, but why? I mean, I don't have to have that level of verbosity in this specification. Now, I don't. I know we don't like this because there were places on the outside and we're trying to have a unified object, but I don't see any reason why it's fundamentally the case that we have to repeat, repeat this stuff when this function is already, you know, deeply tied to this command path.

Listen, guys, I have to go. You can continue, but I have to go.

Yeah, okay. Well, obviously there's a lot to discuss here, but.

Yeah, but. But maybe. Maybe. Right.

I mean, one.

One other idea would be I'm going.

To disappear, but you should continue.

Sure.

All right.

Thank you.

Okay, see you.

Bye.

We can maybe go for a few More minutes. But I was going to say, I'm not sure how satisfying this is, but we could have an object, right, that's like, you know, like a command line argument object, which sort of, you can, you know, includes all this path information, but you can basically query it in a, in a simpler way.

Yeah. Though I will say, I mean, I'm a little uneasy about this overall approach of just having these functions on the right hand side. I'm not sure what the right design is, but one thing I like about this specification, right, Is that if, and I think this is not crazy to assume I screw something up when I'm writing this, right? Like let's say I forget to handle a key or I don't know, I forget to handle a value. Here it's very clear what the fallback is because it's just this replace fails and it goes to my default case and it prints something about, well, hey, there are some arguments you didn't handle. If I screw up something in this kind of specification, let's say this doesn't exist, or I access a part that isn't actually guaranteed to exist, you know, I, I change something in this specification, but I forget to update this handler object. Now suddenly these things are out of sync and it doesn't fail gracefully because this function just.

I don't know. Although, although I would, I would say it's a little bit of a, I mean, maybe if you have explainable pattern errors, it's not a problem, but it's a bit of a.

Well, but there's no opportunity for a pattern error to occur here, right? Like if this doesn't exist, it's just a part out of bounds error. There's no nice.

No, I understand, I understand, but I mean, I'm just, I'm just right. I guess the case would be, you know, I think, I generally agree, but I think that the case for having it be a function is that it kind of forces you to write a pure function out for each of the cases as opposed to, you know, you get back this object and you're going to do pattern matching on it and oh, I hope that I have a pattern matching case for each of the different, you know, parse cases. And it is also repetitive in the sense that up above, when you wrote out your, you know, the parse specification, you list out all these commands and now you're going to list them all out again and just hope that they line up.

Yeah, well, that's the thing. I mean, the holy grail here would be some and this might be too horrifying to contemplate, but you know, where you've got. And I did kind of a experiment with this, but, you know, you've got something like, oh, God, why is this not formatting? Right? You know, something like. And I'm not proposing this exact syntax, but like, you know, that basically. Right. And then on the right hand side you can just use this bound name. So if it, like, if you apply this and it matches, you just, you know, the specification is the parser and it's the comp, you know, thing that as a computation, you get all of it in one.

You know, I just. This is a random point, but this, this would be possible to do with. There was some meeting a few weeks ago where we talked a bit about having a pattern construct with a name like canonicalizing pattern or something like that. And the idea would be that it's similar to pattern where, you know, the, you know, the colon operator, but where you can give it a sort of a function. And. And so the idea being that you can tell it. You can basically tell it. I want you to, you know, I want you to, for example, match if this thing is an underscore string, but I don't want you to actually bind, you know, I want you to bind some other value to this, you know, to this symbol, if that makes sense. Right. So like you'd have something where, right, where we're basically the, you know, there's, as it does the pattern matching, you know, when it matches something, it will sort of run a function and then bound the result, find the result to the symbol, which kind of is what you need to do that here because you need that clap arg thing. You know, you don't want to literally match like you want to actually run something. You need to actually, like what you want to bind to name is not literally just the thing that's inside their place. You want to bind something that's like more processed. Right. I can't hear you, Connor, if you're saying something.

Oh, sorry. Yeah, I was saying, I mean, you don't necessarily need that mechanism. That might be one way to do it. Because, like, in this case, if your specification is actually something like this, your whole pattern is not really a Wolfram language pattern at that point. So the fact that you have to be using pattern name binding syntax is.

Kind of a cheat, I think. Patterns that aren't patterns, I think are very dangerous.

Yes, I agree, which is why I wasn't seriously proposing this, but it would be an easy answer to how do you get the computation? How do you bind the names without using a function where you kind of have to.

That's why I say though, that the. The. I think a construct like that might make it possible for. To do something like this where Is actually a pattern. Yeah, but. Yeah, maybe.

I mean, I would worry that the rules around it, like, you know what is because, like, fine for this specific thing inside. Sure. But then how is. How is clap command matched against, you know, the. The sequence of strings essentially like this. This whole thing is not a pattern either, so it's a little odd. Right. Well, anyway, I mean, I don't know what. I guess this was not.

Yeah.

I think we should wrap it up.

Yeah.

You know, the people who are. This is basically just, you know, ECE meeting personnel. We can kind of follow up on this anytime we want. I don't know how deep you guys want to go with this. I don't mean today. I mean, we could have a working group sort of thing. Or maybe Connor just wants to take some time to integrate the things that I've talked about. Or maybe Connor and Christopher want to work together or something.

Yeah.

Yeah, we can end it.

Live stream.

Yes. Thank you all people who stuck through this very long live stream.