Live CEOing Ep 646: Language Design Review for Compiler-Based Paclets & CompiledComponent

All right, hold on, let's.

Okay. Hello everyone. We are doing a design review about the compiler and Christopher is sharing something here. What are we looking at?

Right, so the main thing we're looking at is stuff with compiled components, which is basically a thing for having compiler based packlets and also just generally being able to. It's basically the package system for the compiler and it also lets you do these sort of packlets whose implementations are sort of, you know, done in the compiler. But there's also this other thing that's I think a much quicker topic, that's this downvalues function thing that I think we should probably talk about first. So. Yeah, so basically. Right, let me see if I can introduce this. So basically the idea is we have this thing called kernel function, which, which basically lets you, in the middle of your compiled code you can write kernel function. And this basically tells it. This is a function whose definition where the implementation of the function it's going to call the evaluator and this is the syntax for it. You say you give it a type and then you apply it and there it goes. So the proposal, and there are a few different parts of this, but the first part to mention is, is to have this thing called downvalues function, where it's like kernel function, except I'm not sure why it's not. Oh yeah, it's just syntax highlighting being strange. Yeah, it's the same as kernel function, except instead of actually calling the evaluator, it will at compile time read the down values for this and compile the.

Transact them in and compile them. Nice.

Right, so the example is here. I've got this thing. I can compile it, it's compiling, I can run it. There it goes. I can do the same with the kernel function and it'll give the same result. But just to prove that it works, I'm going to modify the function and you see the compiled one. It doesn't know about the modification, but the kernel function does because this was compiled before I modified it.

This seems nice.

Right? And so good. Well, okay, so probably going to talk about the name at some point, but. Right, so this is, this is a pretty uncontroversial thing. The more complicated part, but this is the reason this was initially being introduced was basically how you declare, how you write function declarations whose implementations come from down values.

So currently, let me ask a different question first. That's down values. What about sub values and things like that? I mean, what if you had something that you.

I don't Think they're impossible, but they require like, you know, thought and work.

So it's a different function. So we would have a sub values.

It would be something different.

Yes.

The point is that this sub values are not actually functions, right? Sub values are like sub values. Objects. They're objects, you know, which sub values can be. Can be.

Maybe Christopher. I mean, another way of thinking, they are functions, but they just have some of their arguments in a funny place. That's another way of thinking about them.

But anyway, the basic point is this function is specifically intended for down values. And if we want to do any other kind of values, it's going to be a different function in the same family.

Right.

I think if we had, if we had other kinds of values, then it would probably need slightly different syntax. So that would. Yeah, right. So the reason though, why this is sort of important and not just sort of a nice little feature is basically how you define function declarations who get their definitions from down values. So right now this is the documented syntax for doing that. So essentially you have a function declaration, you say call it foo, and then you say typed. And then instead of writing a pure function, you just write a symbol. Right? And then that will get the down values from that symbol.

I don't even understand. I don't understand. Function declaration is as a sub value. Has. Test. Func. What is it?

No, no.

So the at sign.

Doing that sign is with the typed. So let me show you what this, what this? Oh, my keyboard has suddenly stopped working. Oh, now it's back. Ok, that's scary. Yeah. So normally this is what you write, you know, whatever. Right. So normally what you do is you write, here's a function name, here's the type signature, and then here's the function to compile. Right.

So you apply the type signature to the function.

It's. It's a. It's an operator form. You can also write this.

I see, I see.

Okay, fine. Right. So, yeah, it's just, it's just a convention. I usually use the operator form.

It's not a bad convention. I think we.

No, it's fine. I just wasn't used to it.

Good.

Yeah. So in any case, the point is here, is that this is what we do right now is you just give it the symbol and it says, ok, if you give me a symbol as the implementation, I'm going to assume that that symbol is a down values thing. The proposal is to switch it because there's actually another usage of this syntax I think will make a lot more sense. So that instead, the way you do this is you use downvalues function. So you basically say function declaration and then the implementation of the function, instead of giving a pure function, you'd say downvalues function of test func or whatever.

Okay, but that typed wrapper around the down values function is exactly the same thing that you would have used in the usage above, right?

Yes, yes.

Okay, so the issue here is whether that wrapped down values function is. So normally, function declaration is given a pure function.

It's given a typed pure function. What's the foo?

I don't understand. Why is there both a foo and a down values of test func? What is the foo doing?

The foo is the name of the function we're declaring. So see, here's the actual program we're compiling and it's calling foo. Right?

So I have to do with downvalues of test func then.

Sorry, that's where the implementation of the foo function comes.

Oh, I see, I see, I see, I see.

This is the name of the thing, this is the type of the thing, and this is the implementation of the thing. Right, right, Right. So the point is just whether the implementation specific is a symbol or down values function of a symbol.

But in a sense, it's like a module defining a function here. I mean, you're temporarily defining the function foo equals essentially test func with various types specified.

Well, kind of. I mean, but this is. This is just. I mean, this is just the normal syntax or function declarations, like the fact that it's local and so on. Doesn't really matter here. But yeah.

So it's just a function declaration that says this function named whatever in the compiler environment will have an implementation that is this thing that's specified.

Right? So really the only change here. So, okay, the basic point is, here's this schematic of a function declaration. We've got the name of the function that we're declaring, we've got the type of the function we're declaring, and we have the implementation of the function. Right? So the question is, you know, the name spec is staying the same, the type is staying the same. The only question is how the implementation specs.

What do you want to be when it's just a pure symbol though?

Right? So that's, that's this next section. So basically the idea is. Let me see, where's the clearest example? Okay, well, let's just look at this one. Basically the idea is if it's just a pure symbol, it'll assume that the implementation is just another Function in the compiler environment, as opposed to it being a function in the symbol table.

It can know whether it. I mean, I see it. It will only look for things in the compiler environment.

You really don't want it to say plus. And then it says, what's the scoping.

Of the symbol names in that compiler environment?

They're just symbols. So, I mean, it's just if you write this in a package, symbol scoping.

Uses contexts and whatever.

Okay, but so it doesn't have. It's not a scoping construct.

No, no, not at all.

Okay, so there's some globally hanging out there, even though they're ultimately only used in the innards of the compiler.

Well, you write this in a package.

In module or package or something, you know, that's. And in fact, that's kind of what we're doing with the, you know, the packlet thing. But they're just symbols.

So to justify this, though, because the reason why this syntax is very nice is because it means that there's a whole sort of consistency thing where. Right, so suppose. Okay, so here I'm going to declare a function in the compiler environment, right? So here, now I've got some function in the compiler environment called my function. The point is I want to write this, right? So normally what you have to do is you have to write this. You have to say, here's a pure function, here's the input. Here I'm applying the function in the compiler environment to the input, and there you go. But the point is, is that what you'd like to be able to do is just say function, compile, and then just say, you know, here's the name of the function in the compiler environment. Go get me a compiled code function.

It's very nice to do that. What Christopher is saying, it's very awkward to write.

So hold on, hold on. So this is the thing already declared in that compiler environment. The declaration doesn't in fact do compilation. It just says what the. Yeah, okay, so with this thing here where you're declaring it to be a down values function, but now couldn't you also say function compile of down values, function of blah?

You mean like. Like this?

Yeah, I mean, what was.

Although you might need the typed. I think that would make some sense. I. I'm not sure. No, that doesn't work and it gives an absurd error. But yeah, I'll make a notebook to put that in. But yeah, I think that could work. Okay, but I mean, the point is here, it doesn't even matter. It doesn't Matter, this is a downvalues function. This could be anything. The point is just that you can compile the function by name. And the reason why that's really useful, you'll see with the compiled components stuff, which is why I wanted to talk about this first. But there's this idea of a function that isn't implemented but one day would be very nice, which is, you know, has various tentative names, but the idea is basically this is the function for compiling something and putting the results in down code.

Yep.

Right. So you know, this would, this is the most obvious base syntax, like most verbose syntax is. Here's where you, I want you to put the function and here's the implementation of it. So this would install this, it would compile this and install it into my function, the symbol my function. But then the point is, what you want to be able to write is stuff like this and, and stuff like this where you basically just say take the thing in the compiler, compile it and put it in the top level.

Show the syntax of install, the old 35 year old install function.

It's horrifyingly simple.

It's a file.

Well, it is because it uses the.

Intermediate, the template stuff.

Yeah.

Okay, well the template stuff is just how the thing's built. I mean you, you could write a raw thing in theory.

Okay, so this is a binary, this is installing a binary. And it, it in its install actually does what you're wanting to do here. That is it inserts something into the down values.

No, it doesn't. It makes down values, not down code.

No, I understand that, but it inserts something. If you say install of, add to, it will actually assign values to add to of, I underscore integer, J underscore integer and so on. Right?

Yes, that's exactly what happens, which is actually pretty nice.

Well, I'm looking.

This is probably, this is probably something about context in stock notebooks.

This and, and, and. Yes, well, this is a bug. So Aubrey, this is for you, please.

Understood.

I'll report it, have weird quotes around it. Very strange. Okay, right. Well, in any case, the nice thing about this is it, you'll see later. I have an example with a compiled component where it uses syntax a lot like this and where it just makes things extremely simple.

Well, hold on, let's talk about this function here. Okay, so how does this, what, what's another thing that's like this? I mean, we've got. This is essentially an assignment. This is a thing that is meta programming, you know, modifying assignments basically set.

Delayed is Set delayed is a bit like this.

Right, well, so here's the thing. So the reason why, you know, this is in blue because it isn't implemented because it's not for 13.2. And the reason for that is because there's a whole bunch of complexity around basically the fact that right now we don't let people define down code for stuff. And so if we do, there's some stuff, I mean, for example, how do you clear it and where can you find out about its existence? For example, presumably downvalue.

I mean that's what patterns and install. That's what link patterns was for, to figure out what the heck had happened.

Well, maybe then that's actually a reasonable. I mean it's a little antiquated, but it might be a reasonable thing to look at to, you know. Yeah, but because I mean, one of the issues is. Right, so one of the options was maybe it should make a fake entry in the down values table that sort of makes it look as though it has a down value. That's what compiled code function. That's actually, it can't actually have that in the. Because the down values will override the down code.

Another option would be is it has a thing that, that has a thing called call function or something and it puts into the down values a, a, a trivial pattern match whose right hand side is call function or something.

Right, Okay.

I mean, so it's, so it's still going through the pattern matcher, which is not good, but it's, but that's how it does it. So having, you know, a faux entry in the down value seems right to me because then if you clear the thing, it'll clear everything, including that faux entry.

Yeah, you might want a way to restore. I mean there's a little bit more, but. Yes, yes, I think that's all not crazy. So.

Well, I mean there's, there's some down values meta programming we don't really support right now that involves, you know, manipulating, reordering, et cetera, et cetera, et cetera, down values.

I think we do support that if you actually write to down values of foo.

Yeah, absolutely.

You can do that. Yeah.

Okay.

Right.

Well, I think this is not, this is not the time to go into this thing.

Fine, but so let's keep going.

Yeah, the down value function is a.

Thing, but it's incompatible.

So sorry, down value function is what.

So I think the, the, this proposal is to support down values function in 13.2. Chris.

Yeah, yeah, yeah. But the, the only, the reason we bring it up is because there's this thing that changes, you know, changes the current syntax.

I don't. Right now a huge number of people are going to be hanging on that.

Yes, that's what I think too. That's what we know.

Yeah, yeah.

I mean, I think, you know, you might try and put in a bridge message that says something that is not completely crazy in the case where it has a symbol there that it can't find, which is the. Which is what's going to happen.

Yeah, yeah, that can make sense. Okay, Right. Ok, so compiled components. So basically compiled components are. It's essentially the packet. A compiled component is a package for the compiler and. Right. That's all it is. So let me just. Let me just go down this. I have a sort of demo here. So, ok, the first thing you're going to do is just basically append to a component and here the component doesn't exist yet, so I'm just appending to it to. To create it. So if I append to it, we have this compiled component object here it is. You know, I can also get.

It's a little confusing to create it at first by doing an append, because obviously usually append to can't do that. I mean, in other words, that there isn't a create compiled component function.

Right.

Well, the issue is that if there were a create compiled component and if you had to call it, it would mean you have to do really weird things in your package packages, which is one of the problems with the old component system. So in particular, let me.

By the way, if we're talking packages, how come. How come Connor isn't here?

Well, he could be, but also this doesn't actually involve really much interaction with the. You know, it uses like the packlet system, for example, but it doesn't really require any integration from that side.

Right.

Todd's here. I mean, it's a paclit, right? Yes.

All right. Okay.

So the reason. So for example, this is a pack I was going to show later. This is sort of like a grand use of this stuff that works very nicely. So you see here, this is pretty typical. I have the base file and then I have several sub files where the actual component append two is happening. Right, Right. So if. So you see in, you know, this.

File operation M files instead of wl files, but please continue.

Yeah, well, so, so the, the issue is that, See, you know that file gets run when you run needs here. So if you needed to create the compiled component, you'd have to do it here, which is just, just so bad.

I mean, also confusing because the file. Okay, explain what's happening inside the GEOS core base. Wait a minute. Geos core base has, has an append to in it, is that right?

Yes.

And why couldn't it have the create compiled component in it?

Well, I suppose base could, but then that would be. The base is assuming that it's the first thing to run.

Yeah.

If you reordered these, it would break.

Yeah. It forces you to really think about what order what we have here. It doesn't matter what order, anything's loaded. You see, it's just definitions. It's very convenient.

Well, I don't like the name. Then append to. I mean, maybe I, you know, couldn't it just be add compiled component or something?

Well, an earlier name was Register compiled component, but yeah, yeah, but the point.

Is, it's not really what it's doing is it's throwing more things in the bucket that's called my component. Right, Correct.

Right.

And there might be zero things in that bucket initially.

Well, it might not in the design, it might not even exist.

Yeah.

Right. So one analogous thing, by the way, is compiler environment depend 2 but you do have to call Create compiler environment.

Well, there always is one. There's always one.

So, yeah, I mean, you can call it with one argument, but you know, that just defaults to dollar. The point is this will never create a compiler environment.

Yeah. Which is why I don't think the other one should be called append to. I think it should be.

Well, one thing that maybe would change it a little bit is if you run this on some new component that doesn't exist, and this is actually a topic that we're going to talk about later. Right now, this returns a failure, but this could return basically an empty component.

Right.

And so what that would mean is that, you know, it's sort of like all the components exist, they're just all empty until you append to them. That would, by the way, be. Be fairly consistent with how packlets work, because packlets, if you just do something like that. Oh, oh, maybe not. I thought there was something where, where it returned an empty packlet object, but I guess not. Oh, no, it's like this. It returns an empty packlet object.

Okay, so the issue here is you are, you've got a bucket. It might not have anything in it. This might be the first mention of the bucket. I don't know, somehow I, I. But it's probably the wrong name. Compiled component addition. I don't know why that's the right name. It might not be.

It sounds too noun like.

Yeah.

I mean, we could say something like declare, which is sort of a pun because it's like you're declaring the component, but you're also declaring that this thing is part of the component.

That's not crazy. Copy in 1, in 33 and.

Oh, we have. In the old component system there's something called that. But that's okay.

Yeah, we could get. That's. That's not a problem. Christopher. I don't think.

Yeah, yeah.

I mean, it could be declare compiled component as well.

Yeah.

Put a payload in this.

I think that's quite nice. I mean, I think that's. I think declare compiled component is the.

Yeah, okay. I think that that sounds, that's consistent.

With the other declares that you have.

Well, I don't think there are any other. There are a bunch of declarations. There are no other declares.

Yeah.

Is that consistent with the other names? Do you have other names that have compiled components first? I mean, I don't really don't like the verb first style of naming declare file.

Yeah, well, I have. I think it was build compiled component and load compiled component. But they could be compiled component building.

No, I, I don't think they should be. I think that these are cases where like create comes first. I think there are things, Todd, where I agree with you that it goes, you know, noun, verb. But I don't think these are examples of that. I mean, it's subtle and I agree it could go either way, but I think this is slightly better.

It's not easier for searching if they're consistent.

Well, I think it's better to do it this way around especially. Okay, we'll see the other functions that you have around compiled components.

But by the way, on that point about searching, I feel like at some point we should get, I mean, a lot of, you know, IDEs and text editors and things. They won't just search from the beginning, you know, if you type compiled component.

I know, I know, I know. We've been supposed to have that forever and ever. It's been postponed multiple times. It might make it to 13.2.

Right, but that's an argument to not worry about choose function names. But. Yeah, right, but okay, so basically, okay, if you append to it or okay, if you declare to it, then now you've got this component. One thing, this is kind of an implementation detail, but the full form of it is just that. So this is just a reference to a component that is out there. It is not you know, and the reason for this is because otherwise, you know, if this actually had the association in it, then, you know, you, you append to it here and then you have this older version of it sitting somewhere else that doesn't have that declaration in it.

Okay, but when you save it in a file, what happens?

So the way you save. So I think if you, if you just tried to save it in like an MX file, nothing good would happen. But there is a way of serializing it. But that's actually a reasonable thing to think about is how would you, how.

Do you save it with a greater than. Greater than put into a file?

Well, I mean, if you just did it like that, it wouldn't work.

What happens if you reactivate the notebook and things?

It wouldn't work. You'd have to run these again. I guess one thing it could do.

I mean, if it was in a packlet, if it came from a packlet, of course it would work.

Knew where the packlet was.

Yeah. So one option is it could have, although I'm a little skeptical of this, it could have sort of a cache of the association in its second argument. You know, it wouldn't ever display it and you'd still be able to create them like this, but the actual full form of this would have the implementation there. Now the only problem with this is that first of all it might cause some issues with sameq, but I don't think people doing same queue on components is very common. The other thing would just be that you would have out of date versions running around. So you would have something where maybe in my notebook I have a few of these little compiled component things and one of them I choose to rehydrate. But it's different from all the other ones.

Look, it's a bit like the symbol table. You don't by default save the symbol table. This is another indirectly referenced name.

Correct.

Which is an argument to just have it be like the symbol table and not get serialized directly and require that you do the equivalent of a save. I mean, the put won't work, but save could. Could save be made to work with this? I think it should.

This save.

Yeah, that save.

I'm not even sure I've ever used this save. I've used dump save. I've never used normal save.

Well, I mean it saves symbol values. It does the thing that get doesn't do. I mean, get. I'm sorry, put put simply takes an expression and puts it somewhere. Hey, hold on. I'll be back in a second. Keep going. I'll be back.

Okay. But I don't really know.

I don't think it's very serializable. It's like a link object or.

Well, no, the components I think are totally serializable. I mean, they're just. I mean, like, why haven't it come through a packlet?

You don't.

That's fine. It doesn't need to know that it came from a packlet at all. I mean, it's just, it's just a list of declarations and things to compile.

I think if it came from a packlet and then you serialize some different version of it, it would be. All sorts of confusions will happen.

That's true. Like if basically you wrote this component so you're expecting it to come from a packlet, but then you, you also have a serialized version of in your notebook and those two versions are competing, that could lead to weird things.

I mean, the full form could refer to the packlet.

Well, but it can't. Because, I mean, we talked about how there isn't any dollar current packlet thing. Like there's no way of knowing what packlet you're running in.

What do you mean? There's no. What does it mean to run in a packlet?

It doesn't really mean much. I think that's sort of the problem. So I mean, the point being that if you have, say, say, you know, the packlet says in its extensions, you know, oh, I have this context, I define this context, then the point is, you know, well, there's some WL file that defines that context. And you know, the question is, in that delta WL file, can you ask, hey, what packlet am I associated with? But. But there isn't. I'm not sure there's really a well defined way of saying what that associated with is.

I mean, we have this dollar input thing.

Input file name or something, isn't it?

Yeah, I mean, you can capture the location of the WL file at the time it's read in and save that yourself. But if you're just running some function defined in a WL file, there's no way to say where did I come from? You know, there's no dollar file that defined this function thing in the language.

But I don't understand because we certainly have. When you're reading, when you're parsing from a file, you certainly can get dollar input gives you the name of the.

File, wasn't it input file name?

And you can capture that. Lots of people do. They'll define some global variable within their package that captures the location at the time it's read and then manually use that. But if you don't do that, then there's no way just to sort of ask what, what packlet or what input file did this function definition come from while it's running? Unless I see manually.

I mean, we probably could capture that. How would we do that?

I'm not even sure though, what, what it. Because I mean, the connection between a packlet and the dot WL file that we consider the implementation of the packlet is pretty tenuous. The packlet itself is just a packlet info file which says something like, you know, oh, I extend this context. And then when you run needs like, I don't know, I think it's pretty hard to know which. The connection between the actual implementation file and the packlet info is pretty tenuous.

Fair enough. Okay. Anyway, let's keep going on this main review.

Right, well, okay, so the basic point though is that this reference, it doesn't actually contain the information, it's just a reference. So that way, you know, you can't have two versions of them sitting around that are different. Right, right, but okay, so then, you know, you can, you can append.

Entity store is another thing that works like that.

You mean just the entity store header dollar entity stores or whatever it's called?

Well, I mean, entity store has, you know, it is an indirect thing where you have the name of an entity type and then it isn't the entity type. I don't think. Let's see, the entity store maybe does have a content in it, but when you're actually dealing with an entity, the serialization is again, non trivial because you can have an entity of type foo.

Oh, sure. I mean, it just basically that the entity object itself, you know, doesn't. Doesn't contain any of the, you know, it refers to this, this entity store. Right? Yep. Yeah. And if you serialize this and deserialize it, it'll just be like confused. Yep, yeah, that, that is analogous. Okay, but, but okay, so, so here, you know, we, you know, put some declarations into it. You see now it says I have two declarations. So let me, you know, I'll show in a second how you use these declarations. But first there, there are some other fields in the, you know, in these things. So the first one is. Well, right, the first one was library functions. So here basically the syntax is, if you want to add anything other than declarations, you do that with this syntax. So you say the component, then you Say what you know, to what field you'd like to declare something. Because again, we're imagining this says declare compiled component and then you give it, you know, the payload. So essentially what this is saying is when we build the component, which we'll do later, this is just specifying something that we want to be included in the library that gets built. This is basically just saying in that library, you should have these functions defined where this is following the same syntax as function compile, export library.

That last part is. But I don't understand that. You had a thing previously where you said, declare compiled component, and then could you scroll up because you were declaring it with literal code there. Yeah, with the function declaration. Yeah.

Right. So that's declaring. This is equivalent to writing this, if I could type. And what this is doing is it's just appending to the, to the declarations of this component. Right. This is. This is. So the point is, a component is just this basically inert container and it contains information that can be used both if you want to. If you want to basically compile something that uses this functionality, you can refer to this component. It can use all those declarations from this component. The other thing that it can be used for is that you can build the component and then you get this, this built component library, and then that component library can do all sorts of fancy things. Those are the two main things that this does.

But why is this? Don't you have a library function wrapper that represents a library function declaration? And why do you bother to have that second argument of compiled component?

So the library function wrapper is a totally different thing.

Declaration. Oh, yeah, sorry. It's a call.

Right, Right. So this is basically saying. So like, I mean, if nothing else, if I just say. Component, why is my. My autocomplete is being very strange right now? Oh, I haven't run this yet. Ok, so if I say this at all, this might help. So this is. This is everything that's in the component. This is what defines the component. So you see, it's just. It's just this inert thing. There's a list of declarations, here's an association of library functions, here's two things that we haven't seen yet. And so basically when, when we call build compiled component, which I do lower down, it's just all this is saying is that one of the things to include, you know, basically all build compiled component does is it builds this, this library. And this is just saying, by the way, include these functions in the library. That way, when I call load compiled component I. I have access to compiled versions of these functions. I'll basically get back a version of this association where these are compiled code functions.

Why is that person called declarations, not function declarations?

Because they can also be types, they can also be compile expression declarations, they can do whatever you want, any compiler declaration. Right, ok, so let me just keep going here. Ok, so we have some library functions. This other one is installed functions. This is analogous to that function compile, install thing that doesn't exist yet. Right. But it follows the exact same syntax, which is sort of the reason I brought it up. So here I'm writing, you know, here's the symbol name, here's the definition. Another thing that you can write though is you can just write, you know, foo, where foo is something that was declared in the declaration section and that will also work. Although here I don't think I.

What is an installed function? That's something that's going to be put in the down values in the down code.

Yes, down code, yeah.

And I have an example that I'll show in a bit. This is actually kind of the main feature that lets you do nice stuff, but it's sort of hard to explain till we get to the building part. The last thing worth mentioning quickly is there these so called prologues, which are basically just functions that get evaluated at build time, at load time and at declaration time, basically. So the usage of this is for loading libraries. So if I have some functions that I'm defining that rely on some libraries or where when I load the build of the component, I also need to load open SSL or something like that. This is where you do that.

Okay. But it also presumably will give you a progress dialog if the load time is going to be large and stuff like that.

The load time is trivial, the build time can be long, but the load time is never more than half a second.

Even if it was some weird machine learning thing that was giant.

Yeah, like it. Usually the load time is very quick because it's just loading a dynamic library. It's not compiling anything.

I don't even know what loading a dynamic library means. I mean, does it map it into some address space or something? It doesn't even copy any bits.

Well, Tom probably knows more about what.

Basically it has to do a little bit of work. But it. But modern OS is. It's terribly lazy, so it has to resolve dependencies. It can fail.

I love the way that the fact that it's terribly lazy means it's terribly fast. But anyway, never mind.

Yes, yes, yes, correct. I mean, it does as little work as possible.

Okay, all right. Onward.

Okay, real quick, though, just something real quick. In that compiled component, append to, you have prologues, comma, and then some prologues, and you have installed functions, comma, comma, why isn't that install functions, arrow, prologues, arrow. Right. And then you can have a whole bunch in one. In one.

Yeah, I was wondering that too, actually.

Well, so one thing I was thinking about would be supporting some association syntax, because if nothing else, that means you could take the serialized version and feeding it here. The reason why I don't want to just do an arrow here all the time is, if nothing else, because there are things like installed functions, library functions, where if you give only one of them, it's a rule, right? So here I'm giving a whole association of them, and here I'm giving only a single one. And so it makes sense for this just to be a rule. And so if you allowed for their weird precedence. I mean, this just looks really weird to write.

Well, but if you made it an association, so library functions, arrow, association.

And.

Then you can have as many of these things piled into one append to as you want different types of components.

Right? But you could also do the different types of components. You could also just support something like, you know, you could support the. I guess it would be a little bit inconsistent, but you could support that and then also just support this.

I agree with Todd that I think it'd be better to have these arrowed things. And I think that this is not. You know, these things will be as often written by machines as by people. And the less obscure this thing is when you see it, the better. I mean, I think.

The one thing about the pure rule form is they kind of look like options, right? You know, the last argument to a function, why not just. Yeah, but you could group them in a list or an association, or you could just ignore the fact that they look like options. Just use a straight rule.

Well, I'm worried that you may need options, right.

If we need options later, right, Then we would have a problem.

So another thing to mention quickly, but I'm not sure this really affects it much. Let me show you a real example, because you often end up with.

Right.

You often end up with long. You know, the inside of here is something big. So here's an example where I'm appending declarations to geoslink. This is taking advantage of the fact that the second argument defaults to declarations. But basically, I'M giving it this list but you can see this is a long block of stuff which sometimes I think can be awkward if it's on the right hand side of a rule.

Maybe. I mean you just put the rule up there and then have an arrow and then it hangs. Then all the rest of it is nested. Doesn't seem that terrible.

Would it still default to declarations though if you just give it a list or a single item?

As long as that isn't ambiguous. I think it should. Yeah. Why not?

Trying to think if there are any cases, I guess there are no. There are no declarations that take the form of just a rule. So I think it is unambiguous. Yeah.

Okay.

Well I can.

I can look pretty nice by the way that, the way that whole declaration.

Thing, you know, Incredibly nice.

Yeah. I'm going to show this example, the big example at the end. So this is spoilers.

Okay.

But. But yeah, okay. So fine, that can, that can use that. That could use rules. I think that's, that's fine. But now also, would it support this? I'm not sure how common this is.

I don't think it needs to.

It's not so terrible to put that in an association.

Yeah, I mean if you write like this, I think it looks much better to put it in association than it does to have it just sitting there.

Let's keep going. I think probably the association. I would define the most encrusted form first and you know, crossed. What's that?

Decrust later.

Yeah. Yes, it needs to be smooth. Okay, keep going.

Right, Can I just jump in one more thing because did we decide that Declare compile component was the replacement for append to. The thing about Declare compile component is it sounds like it's this. This updates an existing compile component. Right. It really is a kind of an add to thing. It looks like a overwrite kind of a replace kind of operation, doesn't it? Declare Compiled component. There's nothing in that name that suggests it's adding to something.

Well, maybe Compiled component declare might be mildly better in that sense. I mean it's like.

Well, I like that better for other reasons, but I'm not sure if it changes the. My thinking on the.

I think it does slightly.

Well because Declare compiled components Very clear that the thing you're declaring is the compiled component as opposed to Compiled component declare which where it's a bit of a pun about whether you're declaring the component or whether you're declaring things that are associated with the component.

Yeah. I suppose, but I don't know. I prefer the verb now.

It's not going to work because of the initial.

Yeah.

It's very hard to have something which both represents, you know, if there was nothing there, start it and add to what's there.

I mean, like register and things are kind of. Are suggestive of that, where you're just sort of adding into some state.

Which is why register compiled component was another possibility.

Yeah, right.

It was in the original thing. It was.

But.

What do we use register for elsewhere? I mean, we have register for entity stores and things like that.

We have register for packlet sites.

Does it mean the same thing?

Yeah, it means add to a list of known packlet sites.

It's like a little different. No, I know, but yeah, it's.

It's right. This is more like adding to this compiled component. It's not registering a compiled component in some.

Right.

Global data.

Okay, we're gonna run out of time, so let's keep going. We got.

Yeah, okay, let me, let me go to the examples, but. Right, okay, so let me skip down here. Ok, so the first thing is how you use components, other compiled programs. So we have this compiled component. One of the things that defined was this myfunc thing. So, you know, if I compile this, myfunc is defined, but you can use compiledcomponent as effectively a declaration. So this is the slot where you'd put things like function declaration, type declaration, whatever. You can also just use this and it effectively, you know, and you could put it in list, include other declarations too. But this effectively is just a big ball of declarations when you use it in a case like this. So here, you know, myfunc is defined similarly. If you wanted to include this in a compiled component, then, or sorry, sorry, in a compiler environment, you'd also just run compiler environment, append two and just like any other declaration, you can throw this in and it will, you know, put it into the declaration.

So is the content of a compiled component always declarations or can it be other things?

It has other things too, but declarations are the only thing that's relevant for doing compilation.

I see.

The other things are relevant for sort of interaction with the top level. Yeah, with the library. Okay, so yeah, so basically just it behaves like a declaration in all these contexts. Ok, so the libraries though, are the big. The big thing. Ok, so there are basically two functions. Build component compiled component, and load compiled component. So if I call build, it will go off and it will compile it. Ignore this message. Just Because I'm using some internal version of the packlet. You see, it ran the prologues that might have, you know, loaded the libraries and so on, and it returns a success object and it tells me, you know, where it was built and so on. By default it builds into like that place in the user base directory where create library dumps things. So it kind of just gets dumped in this central area that find library can find, but which is sort of, you know. Yeah, like a scratch space area. And now to load it. So there's, there's, there's. You can.

Sorry, that. Presumably you have an option that puts it in a different place. Is it a, like.

Yeah, it's a second argument and it's a little bit below. In fact, the, the, the most useful thing is you can say, you can give it a packlet object in the second argument and it will put it into the packlet in the correct directory and so on. But, but okay, so, so the, the thing that this does, though, so you can load it explicitly by calling load compiled component. But I'm going to show a different way of loading it first, which is that when you call, you know, declare compiled component and you declare some installed function, when you do the declaration, it actually sets a down value for that symbol. So above we ran somewhere, we said, yeah, installed function. We added this installed func one and when we ran this line, it added a down value for installed func 1 that looks like this. And I mean, actually this could probably be a bit cleaner, but basically this just loads the compiled component and then tries to call the compiled version of this. So this is, this is very analogous to how packlet auto loading works, where, you know, the symbol will have some sort of definition that's kind of a placeholder, but once you use it, then it will actually do loading. And so if I then run this, you see it, it ran the, you know, it loaded the thing and now it's defined. And now when I run this, again, this is just calling the down code definition of this symbol.

Why is that thing blue?

I'm not. So it seems to be inconsistent about whether it's blue or not. It's blue because there are no down values for it, because it just has down code. It seems when I do this, when this is defined in a packlet, so it's in some context, as opposed to global, it doesn't turn blue. And I don't really know why.

Yeah, it's something to do with the syntax coloring, but I'm sure that can be fixed.

Okay.

Yeah. But the point is this is just a down code function now, right? And so alternatively, that could have been loaded by calling loadcompiled component, which does the same thing, but which also returns other stuff. So for example, it returns the library functions, which includes all those library functions that you asked to be compiled. So, you know, that way you can ask for things to be compiled and you can access them later through this. And you know, it includes whatever it happened to install and stuff like that. One other nice feature is if the, if the component that you have in memory gets out of alignment with the component that it loads, it can, it can let you know, which is sort of a nice feature. Okay, so let me maybe show a packlet example because it makes it clear what's going on. So actually, you know what? I'm going to show the fancy packlet.

Show the nice one.

So.

Yeah, yeah. So it's. Okay, so here's, here's. This is a fresh kernel. This is a different kernel from this one. So this. Right, so. So here actually, let me make sure it's a fresh kernel. So I have this pack of GeoSLINK. It defines a compiled component called GeoSLINK. And for some reason this notebook is hanging. It's just. Oh, an hour back. Okay, there we go. So now I've. Previously it's very strange. Ok, well, previously I built this. It's not even actually not responding. It's responding just fine. So I don't know why it keeps thinking it's not responding. Okay, I'm gonna, I'm gonna reopen this because it's just being broken.

This is a very streamlined way to get significant functionality from. Built with a compiler.

Yeah. Okay, so basically what I'm going to do is I have this packlet, it's already been built, I packlet directory, load it, and I run needs for a normal packlet. That's all you run now, in order to do it, I don't have to run any kind of load the packlet, anything like that. I'm just going to call Geos Initialize, which is just a GEOS function. This isn't a function that I define. This is an installed function that calls some C code, whatever. And the point is, is that GEOS Initialize, you know, when I run it, it will automatically load the library. And now all these functions are defined and I can do whatever I want. And the only reason, just, just to be clear, because I just realized this, this might be misleading. Geos Initialize is not something that like loads a component. This is, this is Just a GEOS has a function that needs to be run. Yeah, so basically you could have put.

In your some kind of prologue if you wanted it to be auto run whenever any other function from the GEOS library is run. Or is that not true?

No, you couldn't put it in prologue. Maybe we, I mean I actually, we had discussion of maybe having a thing called Epilogue which might be able to do something like that, but I have to think about.

It seems useful to be able to have because I think a lot of libraries, a lot of external libraries will have something that requires initializing them if you're calling a function from within the external library. Like this GS boundary thing.

Right, Let me just make a random note about that. But. Right, so the point though, the main point here though is that from the perspective of the packlet user, it's as if these things were defined with set delayed. You know, there's no visible anything that, that these are, you know, involved components.

Can we see the source code of the, of the like from GEOS and things like that? It's over there.

Yeah. So, so basically all you.

Declaration. Yeah, go ahead.

This is just the list of declarations. So basically I'm saying, you know, measures, saying okay, penned to. I want, you know, all these declarations. And so these are things. This is just standard compiler stuff. I'm saying I want geosarea. Here's this declaration a little bit lower. I have a function declaration that defines a function called geosarea and it's got some implementation that calls the library function.

What's that exception code thing.

This is an unfortunate thing. This is basically just because we don't have an exception mechanism in the compiler right now, or at least not a documented one. Basically because there isn't a top level exception mechanism to emulate and there's enough talk about maybe having something like that from the error handling stuff that we don't want to invent.

1.

Fine, exactly.

So, yeah, so this is sort of in limbo right now, but, but okay, so I'm defining all these, you know, all these functions, something like the name dist pointer.

Is that a C language name inside the GEOS library? Or what is that?

Where do you see dist pointer a.

Little bit higher up or G1, G2, whatever they are.

I mean these are just argument names to the function.

Oh, I see. Oh, I see, I see. Okay, fine.

GEOS disjoint is the function in geos. Yeah.

Right.

So in any case, I declare all these functions and then I just say installed functions. Bang. And There's a big list of names and then when I call build, it just knows, okay, I'm just going to build all these functions and put them in the library. And then when I load the thing, anytime I refer to one of those symbols, it will load it and it will have the implementation there and everything.

This is very, very nice, but I mean, so within a packlet, a typical thing would be that those symbols will wind up in some fairly private context that can then be sort of fished out of for various uses.

Maybe they might be the exports from the packlet. They are, look at the packlet info here. They probably are in this packlet. Maybe they're.

I don't think I mentioned them, but they're mentioned through the context.

They're exported. These are the exports. Oh look, look in the geoslink dot M then maybe there.

Oh yeah, sure. They're listed here. Yes. Right. So I mean, these are just exports, right?

These are the exports from the packlet. They're not private. This is, you know, it. And going through down code is so frigging fast, you know.

No, I understand that. I'm just saying that in terms of what happens when we're writing code. I mean, you don't want to spray these things into the system context or into the global context.

They're in a pattern.

No, they're not. I mean they're just. These are just symbols. There's nothing special about them. When you call, you know, one of these, one of these things here, it's just like calling set delayed. It's just a symbol in whatever context.

In whatever context it's naturally in.

Yeah, correct. Yeah, it's not doing any parser wizardry or anything like that.

Well, so I mean the real thing here is the translation of data structures, which I suspect is non trivial.

Right. I mean that's what, that's the big file. That's, you know, the big file which actually does the conversion between a geos multi line string and you know, a line expression.

Does it actually work? Does all this stuff actually work?

Yeah, totally. I have a great example of this. I mean, I think that the eventual use of this particular package is. I think Jose talked about wanting it for geopolygon operations. But I had an example that I was working on just like a couple days ago where I had like 72,000 census tracts which have like millions of vertices or something. Actually not millions, but it's like, it's like 100 megabytes of geometry or something like that. And I was doing things like, you Know, computing the region union of, you know, 20,000 of them and it runs in like half a second and like region union, that takes like hours. So, you know. But the other important thing is that I had operations where I'm passing these things back and forth many times. And as a test I tried making it serialize and deserialize at each step, right? So basically it goes through this code of turning it into an expression to turning it back into a GEOS thing at each step, even though there was like 100 megabytes of geometry. It does that, you know, that, that back and forth is negligible. Like it was like took, you know, 0.05 seconds or something like that, doing it like five times. So it's, yeah, it's, it's very, very efficient.

That's great. That's great. So I mean, look, this is, this is now, you know, this is becoming a major use case of the compiler is library linking, right?

And then the whole point is with compiled components, it'll actually be, you know, there'll be a way of making a packlet that does the library. Because the issue right now is, is that we don't, you know, without compiled components, there isn't a good way. If Jose says, I want a library, you know, I want a way of calling GEOS with the compiler. I can give him a notebook that has code that you can run, you know, but. Or I can give him a SO or dot DUI LIB file that he can call library function load on and get back a big association of functions. But this is the only way to be like just call needs on this and you have a bunch of symbols, right?

So you can include it in a packlet and so in the packlet repository and in the function repository, it becomes possible to just incorporate functions that make use of external libraries like this.

Yeah, yeah. Now, there's also some interesting stuff. I mean, we haven't really thought much about this, but for something like the packet repository, we could imagine having some kind of service for. Basically we run the build compiled component, right? And we can run it on all platforms and stuff like that. But I haven't, I haven't, you know, I'm just mentioning that. But we haven't actually done any work about that, so I don't know how possible it is.

Well, I mean, we have a system, you know, there's a slightly older version of this component system that we've been using for several versions, like to create libraries for the kernel and they integrate really well with our team city you know, that's. That's all been figured out. So that. That would be a way.

Right.

I have to disappear in a moment here. But, but.

So what do we do? What, what do we do with this? Because it's very nice.

Well, I think we get to.

Yeah.

Good, good.

And I mean, it looks pretty clear. I mean, it looks, you know, it's somewhat complicated, but I mean, how could it not be? I mean, correct. Given what it's trying to do? I. I mean, I think the. The big thing internally to the company is, I mean, I know we've been looking at various libraries and people have been spending ridiculous amounts of time integrating those libraries. It's now kind of. This makes it, you know, we should be figuring out what libraries people want to integrate and making sure that all this works smoothly with them.

Yep.

And I see a question on a live stream about WebAssembly and GPU, TPU stuff and so on. That's a different direction in the compiler also being worked on. Correct the. All right, looks good. Okay. Anything else we need to cover?

No, I think that's good.

I don't think so.

Okay, good stuff.

It's very, very, very nice stuff. I mean, and this is very impressive. I mean, I don't know how long this GEOS link thing took. This looks like. I mean, was this fairly straightforward to do?

The main part of JSON, like all this conversion code and being able to call arbitrary functions took about a day and a half. And then I've since then done little things of just adding little functions. But adding functions is like trivial. Like for example, the other day I added these prepared geometry functions and it's just like you just write the declaration for the function in C and then here's just a trivial thing. It's just converting. If it returns a zero, that's false.

Right?

I mean, it's pretty simple. So I mean, the only hard part was writing the conversion code, but again, that took like a day.

So what kind of documentation do we have for people to understand? What on earth to do here?

Do you mean how to use the compiled components or how do you know.

How to take a random library? And you know, like this, there's a good. There's a.

There's a. There's a tutorial that I wrote in the last version that has a simple example with OpenSSL, a much more extended fancy example, SQLite. So this is the place to go. But of course this doesn't use compiled components. I think we'd want another tutorial explaining.

Different one similar but.

Yeah.

Yeah.

Looking very good. Looking very good. Yeah. I mean, the. Terrific. Yeah. By the way, Christopher, are you in the. There's a meeting later that is, I think, is finishing Astronomy and Geography. You might. You might want to join that.

I have some other meetings later, but I can. I can try and join.

Okay. All right. Sounds good. Okay, see you all soon. Thanks a lot.

Okay, bye.

Bye.