Live CEOing Ep 710: Language Design Review for Foreign Function Interface Functionality

Okay, hi, everyone. We are doing another Wolfram language design review for the 13.3 release. We are talking about the foreign function interface mechanism. And in an attempt to remember what we're talking about before, let's take a look at this. Ok, that's more detailed than. Ok, so let's take it from the top.

Okay, so. Okay, so basically last time we renamed a few things about foreign function load and foreign function, et cetera, et cetera. That's all been done. We also, the main thing we did was before there was all these, you know, buffer to list, list to buffer, string to buffer functions, et cetera. That's now all been moved into these, what, like six functions for import, raw memory, export, raw memory, read, write, raw memory, allocate, and free raw memory.

Actually, it's import and not read raw memory.

There's also read Read is the lower level, lower level one that lets you, like, you know, read from an offset. Like if you're reading from an array, import is the higher level one that you want 90% of the time.

Okay, so let's see what this does here. So foreign function load RAND bytes is the internal C function. This is the path, this is the signature. Is that correct?

Yep.

That seems very lovely. And returns a foreign function. Okay, now if I want to run that foreign function, what do I do?

You just apply it like sub values.

OK, so if I take percent 3 and I sell rand rand bytes and I. What does it take? Let's see. It takes a raw pointer. A raw pointer, right?

So look two lines down. Okay? So the next line is generating the buffer that you need to feed into it. And the. Actually, you know what, let's do a more minimal example. Just if you want to do the simplest thing just on a new line, and maybe you can try and write it yourself. In compiler demo base, that's the name of the library quotes. Compiler demo base, there's a function called add1 that just takes an int and.

Returns an int demo based like this.

Lowercase C. But yes, B, C.

Compiler.

It's somewhere there's a compiler, demo base, dylib.

Okay, all right. And that is a foreign function of some kind?

No, no, that's a library, like run find library of that.

Okay.

Okay, so you're going to load a function from that called add one.

Okay, so I say foreign function load from that thing. Is it like this? How's it spelled?

All lowercase add one.

Okay, let's see.

It takes an int and returns an int.

Okay, so it's just list C int does it need the list on the left hand side?

Yes.

Okay, there it goes. Okay, so now I say percent of 89. Whoopee doodle doo. That's lovely. Okay, I get it. So now for this one I have to actually deal with pointers, right?

You have to make the arguments. So in this example, so allocate raw memory. It has either be given a single argument which is just a type and it will allocate one of them, or can allocate more like here.

So question, what does this mean? So should this be. I mean, is really. Is managed object generically the right term?

Okay, so this is one of the things. This was going to be the second or third thing that we talk about. So there's a little bit of complication around. So it's okay. Basically this is the right thing for it to return almost all the time. There are some cases where you don't want the managed object right now, although I think this should probably change. There's an option called Memory Managed arrow true false. It defaults to true. If you set it to false, it returns just a raw pointer without the managed object around it. This breaks the rules about options not changing the heads of the return type the return. So there are either some quite complicated options that involve essentially duplicating the same behavior, but hiding a bunch of the memory management inside of the pointer object, which the more I think about it, the more I think is a bad idea. Another option is to just get rid of the memory management option Functions like allocate raw memory would then like always return a managed thing.

Why would you want the raw pointer?

So if I have already.

Yeah, if I feed RAND bytes, if I want a raw pointer without management.

Right.

Why would I want that?

If you for example, are, you know, handing over ownership of it to some other code. Right. Like if, for example, I'm maybe I'm allocating some memory, but it's going to be essentially owned by some external program for a while. And you know, I don't want to necessarily have to keep references to the original expression around.

If you were using wstp, you wanted to pass the memory to wstp. You might want the actual unmanaged, but.

Why not just pass it as a managed object and WSTP construct the management?

Supposing the WSTP didn't do that? You're right. Yes, but supposing it didn't?

Well, no, no. I mean most all the functions in FFI know, FFI are fine.

Yeah, absolutely. That's why it's rare Right.

The more important case, the case when you wouldn't want. When you'd want an unmanaged pointer is like, I'm trying to think of an example with a real library here. But like, you know, there are situations where. Right, There are situations where, for example, you have to allocate memory, but then the library is going to tell you when you have to free it, as opposed to you figuring out when to free it by reference counting, because this is just reference counting expressions in the language.

I see what you're saying is there could be external. This is management in our ecosystem, you're saying there could be external management, which in fact. Where you in fact don't want. Even if the reference count went to zero in our ecosystem, you don't want it to free because it actually has an external tentacle, for example.

Yes, but.

So why not have part of the managed object be something that says where are the tentacles?

Because it's like, in general you just can't know, right? I mean, like, in general there can be arbitrary tentacles. Like, that's why we have free raw memory, which is the function for freeing it if it's not already memory managed or it's not already in a managed object.

Well, so may I make a suggestion? Maybe what we should do is the option to allocate raw memory could contain something that says it is a managed object, but by managed it means it still might have a tentacle missing and therefore don't free it.

Well, there is a free known as.

Not a managed object. Well, fair enough, but here's one of the ideas right, from like the last couple days is habits that allocate raw memory always returns a managed object. But then if you want an unmanaged object, you use the function unmanageobject on it, which is an existing function for compiler only stuff, but which we already talked about using for this. And basically what that function would do is it. It diffuses managed objects so that, you know. Yeah, they'll no longer actually free the thing.

That sounds reasonable to me. And a reasonable name even for it.

And that function, I assume it returns the inner value, right?

It doesn't definitely.

What is the difference in a value and a payload?

This is a kind of subtle thing. So, for example, take that if you say, right, basically you can have two managed objects who sort of represent slightly different values, but which need to share a reference count. So here's an example that's a managed object to a raw pointer of unsigned intra 8. If you ever have a raw pointer, you can always Run RAW pointer around that and give it a different type to essentially cast it cost. Yeah, so try running raw pointer of out, comma, you know, some other type other than unsigned integer 8.

I don't know, let's say real 32 or something.

Yeah. So the point is that this goes, you know, through the managed object and it cast the internal thing. But this still, like this, has to share a reference count with the one.

I understand, I understand, I understand. Is payload the right term for that?

I don't know.

Okay, hold on. Let's think about this.

So I presume this also just to add a case, if you were to do something like, you know, index eight bytes after a particular pointer.

Yes. Yeah. That. That would be the value.

Right. Now we don't have a function for doing that, but. But we maybe could.

The value could be that. But the, the. I mean, to me, this would be. Rather than calling it payload, it would be RAW data or something. It would be. Is this.

So.

So I think it's more like. It's more like the root of the ownership tree or something. Right. Because you might, you know, you could do this multiple times where you're indexing, you know, eight bytes ahead, eight bytes ahead, eight bytes ahead. But there's still something that's like. Ultimately, what was this derived from?

Well, another way to the way this actually works, if you run input form of that, although this is a little bit broken right now, basically the way this actually works is that the managed object. Oh, okay. Never mind. It's too broken to even see right now. Basically there's actually two levels. There's the outer managed object which contains the value, and then there's this inner managed object which contains the payload and the reference count. And basically you'll have many outer managed objects with different values that all refer to the same kind of inner object. That makes sense.

Well, I would just call it. Then I would call it raw object or something down at the bottom. Under, rather than payload, it's the managed object. Right. There's the outer wrapper, and then the thing underneath is the raw object or the base object.

Yeah, I think. Yeah, I think base. Something along those lines better captures what that actually is. Because what you really want to understand when you look at that is this is a derived thing which is derived from some other thing which you have to reason about when you're trying to figure out.

Sounds reasonable to me. Christopher, what do you think?

That's fine. Yeah, I mean, it's basically. Let me see. Right. I mean, the thing that's like the value of the managed object is essentially, you know, the value that it represents in the sense that, you know, you would imagine that, like, you want to be able to treat this managed object as if it were. As if it were its value. And then this other thing that. What we're now calling the base object is the thing that actually that the freeing function gets applied to when it's time to free it. So I think.

Go ahead.

Visually, I just.

Oh, sorry, yeah, go ahead, Connor.

Well, I also visually, like, the freeing function doesn't group with the value, it groups with the base object. Right. Because that's ultimately what that freeing function has to get called on, I assume.

Yeah.

Yeah. I mean, I wonder if that shouldn't be some. If that shouldn't be one field that's, you know, base object that inside of it has the freeing function.

And, well, you want to be able to display the freeing function here. That's very useful. And the very fact that it has the freeing function tells you something about what the hell this thing is. I was thinking that what we could do might be to not have that label value colon, just have the managed object and then it would line up nicer there, just have the manage object just say managed object of raw pointer and then freeing function, and then it's got the other stuff.

Wait, you need to get rid of the value colon later.

Yeah, because I don't think the value colon is doing much for us, I guess.

I'm not sure if summary boxes support that, but I can check.

Well, why don't we see if that can be done? Yeah, if it can't be done, then never know.

String, I suspect.

Yeah. Well, yes, that is a hack.

Would it still. Still show the colon, though?

No, no, the colon. The colon and the space you have to write yourself. So no.

Oh, okay.

Well, then we're in business there. Okay, This is nice. Otherwise, I mean. Okay, can we keep going? I mean, this looks good to me.

So you can see if you. There's an example piece of code which is analogous to examples that you probably don't remember. The generate random bytes. So that's like if you packaged everything up. That's.

I remember that. I remember that. I remember that. We went through it. And this is much cleaner.

Right. And note the import RAW memory also. So basically it's. It's somewhat analogous to any of the other import export things. It takes the format in the second argument, although the format is nothing to do with the import export format list, it's its own list, which you can find in the docs.

I'm sorry, excuse me, what did you just say? Could you.

I was just saying that the second argument of import raw memory is not an import export format like the other ones. It's its own list of import export.

Formats formats which you can find in the docs.

Some of these docs are more real memory, not bytes. Raw memory? Yeah. Does that work? No, you have to use the docu tools. There you go.

Okay. Looks nice. Why can't you do. I mean, like, if you wanted to import a struct or something, can you do that with types?

So, so, right. So, so the, the pointers are always type annotated. So if it was a pointer to a struct type, then you could actually just use read raw memory, which assumes that it's, you know, an array of one element kind of thing. Yeah.

So these are using what are effectively top level types, whereas read raw memory is effectively using what. What are effectively compiler types. Is that right?

No, like it's, it's. So, for example, import raw memory with list is doing basically read raw memory, except it would do more than one of them.

What's the right way to think about whether the type is specified in the object versus in the second argument to the importing function?

So the right. So basically the pointers that you feed to import are always type annotated. So they always know what basically the element type is. Because these are basically all array types. So the format for import raw memory is only telling you what is it being converted into. Like it's telling you what's the Wolfram language representation you want to get back, not what is the kind of thing you're pointing at. Okay.

And presumably there's some table there mapping from, you know, if you try to import C struct as a byte array, you get an error. It knows that it can't do that conversion. Okay.

Yes. Actually, for some of them. Yeah, for some of them I intentionally didn't have some error handling because there are cases where, for example, I mean, arguably you can just do casting where, you know, you might say, oh, well, I know it's a. I know it's an integer, but I just want a byte array anyway.

Yeah, I wonder, I mean, is that I'd worry about things like padding and, I don't know, offsets kind of screwing that up in weird ways. That might not be obvious. I wonder if it shouldn't do error detection by default, but perhaps have some, you know, some kind of allow casting option that you could set to true. The False to. False to disable. That kind of error checking. Yeah, like suspicious cast checking maybe.

Yeah. I don't remember exactly which things had error handling, which didn't, or like which things had type checking essentially. I think if any of them don't, it's basically just the byte array one.

Sorry, you're saying can this autocast can import RAW memory? Autocast.

Right. The question is, if I have a pointer to an int and I ask it to be, I'd like, you know, import it as a string. That should fail because it's like, well, I only know to convert char stars to strings. I don't know how to convert int stars to strings. Right. But if you say something like give me, you know, here's an int star, give me a byte array of length 10, then it is unambiguous what you've got to do.

I don't know why you can't do it. I mean, you could import, I mean, you would, you would import, you know, because it's a null terminated string, it's not going to even terminate. It's going to start snuffing in random pieces of memory.

Yeah, I think this is a fairly low stakes question, but yeah, I think either way, I think it's kind of fine.

I could be slow today, but I'm curious what would. If you have like an INT star as the memory and you did import raw memory of byte array, 10.

I think it does a byte array of length 10, not 10 integers worth of bytes.

So if like an integer is four bytes, it reads the first integer, the second integer and the first two bytes of the third integer and that's what it returns. Okay.

Depending on its endianness, it will get whatever it gets.

That's why there's RAW in the name.

Right? Okay, that makes sense.

Does. Is there any sense of endianness, by the way, in any of this stuff? Of if you're reading across. I have no idea.

I have no idea.

What's that?

Frankly, I have no idea.

Yeah, okay, fine. I mean, because we have an endianness, don't we have a variable that represents the endianness in.

Read, you know, in some of the binary read functions, but typically it's only in bitcasting between things of different sizes that really matters.

Yeah.

If you bitcast an array of ints to an array of bytes, you'll see that difference. But if you read them sequentially, then it, you know, figures them out.

Does it? I mean, for an integer. Well, anyway, never mind. It's a detail let's not worry about it. Okay, let's look at the second example here.

That example is quite similar, it's just a little more complicated.

Okay, this is. The whole raw pointer thing is using.

Export raw memory though, which is nice. So export raw memory is kind of similar to import, except that. So first of all, it can automatically often determine what it's doing. Like if you give it a string, it'll assume it should turn into a null terminated char star.

Sorry, I have a very detailed point. The fact that this doesn't wrap and just sort of truncates in this weird way. Is there any way we can do better than that?

We could use the better syntax for colon, you know, type specifier. That would make.

Well, right, that. That's a different thing, which I think we should do anyway, which is just that. Right, right now we have a nice import form for type specifier, which you can see in the import there. But it doesn't format like that in output, which it really should.

Oh, it should. We should just change that. We really should just change it for input form, everything. We should just change it to use that. I don't think anybody wants to see the type specifier version. Does that make sense? Let's just go ahead and change it, even down to the level input form and that will vastly improve it might.

Have some hairy consequences for 13.3. It would be better to do it immediately after the code, you know, in the next version.

Well, look, at least for this output.

It'S not a problem that could be done differently, I think.

Yeah, okay, fine. Okay, so hold on, you're saying this has a raw pointer which returns a raw pointer here, and it takes a raw pointer and then returns a raw pointer?

Yeah.

So that exports raw memory, makes a managed object, that exports more raw memory, makes managed object, then this runs it. Let's watch it happen.

I didn't run the form function. Yeah.

Lovely. Cool.

And now if you import the RAW memory where it wrote the buffer, then you'll get back the hash.

But what about this thing here? What's that?

Oh, that's. I mean, you have to look up the docs for OpenSSL to know what that is. But I can tell you, I think it's just the buffer again. Yeah. If you look at the pointers, it's just the buffer that you put in again.

May I make a suggestion though? Could you not have some way of giving documentation in this foreign function thing? Just some basic. Does that make any sense?

I have no idea what you mean.

No, I mean to have in the summary blob. Why not have descriptions of these arguments that you can optionally give that appear in the summary blob or. Because otherwise it's like, where do you go for this documentation? Okay, let me make it.

You have to go. I mean, like, right, like foreign function.

Load could support something like annotation or labeled or something along those lines on those.

Why don't we do that for Wolfram language functions?

Because we don't have an external wrapper that. That loads them.

I mean, like, why can't you annotate the. The arguments in your. In your set delayed. You probably should be able to.

Let's. Let's have a discussion in ILB about that. You probably should be able to do that.

Okay, careful with your hypotheticals, Christopher.

But, but in this case at least getting a something that is like what we have for resource function where, you know. Because if I do a resource function or a NET NET model, let's say, you know, of G. I mean, I.

Think the thing is though is it's just. I mean, you know, realistically, the thing you need to do if you want to understand this is read OpenSSL's documentation.

I know, I know. So I'm making. Gonna make a suggestion. See this here? Oh God. We have got to fix this thing where it escapes. Do you see this? I move down and it runs away. Whoever's project managing here, remind John Fultz it's a top priority thing to fix this. Because I don't know how to chase the raft, right? Everybody's going to have drowned. Nobody's going to be able to find the raft.

Or the tongue or whatever it is.

Yeah, this one is a raft, I think. But okay. The main point is that I button opens external documentation.

It's not very external though.

This one is still out. I know that, but why not just provide a URL that it can open just so that you're not left.

So the thing is, I have no idea where you'd ever get such an object, if that makes sense. Even if you were parsing header files like, look, I'll post in the chat, there's the relevant documentation. Right. I don't know where you'd find that URL. It won't be in the header files. It won't be, you know.

No, I understand. It's. You as a human have to find it.

Right, but that's like the amount of. I just think no one will ever do that.

Really.

Yes. Like I've done. I've done examples with this, like you're importing, you know, dozens to a hundred foreign functions and making a big link like this. You're not gonna, you know, and you have to be reading the external documentation anyway, so it's really not that useful.

Okay, all right. I mean, you could make a link to. It's just like where does the eye. Is there an I button? And if not. Well, there isn't an I button here.

Yeah.

So. Okay. I mean, it just seems to me if there was an I button, you could link it off to one of these, you know, to.

It just feels like. It just feels like. Oh, if you're calling, you know, I don't know, you're calling import from, you know, your URL. URL. Reading from an API. You know, maybe there should be some way of. Of getting the API's documentation. It's kind of like, you know.

Yeah, right.

You just have it open in another tab. This is how you do it.

Okay, fair enough.

Any other system, more complicated way of doing that.

Where did this come from? And so on. Whatever. I'm not good.

Well, it does know about what library it's from. If you press the plus button, it might probably even says. Although it might be a bit ornate. Yeah. Lib crypto.

How do I see that?

The library field.

Oh, at the end.

Yeah, actually, I think it does if you give it a shorter. I'm not sure if it. Actually, no, I think it. That is post find library. What it shows there is always post find library.

Okay. Okay, let's keep going here.

Yeah. Well, so that's.

My point is not as completely stupid as you're making it out to be that there's some kind of URL that can be associated with a foreign function which just has you. Gives you a fighting chance to be able to find out what the hell it is.

I mean, the thing is, even. Even popular libraries often don't even have like documentation like that on the web. So it's. It's. Yeah.

By the way, was Stefan responding to the. The escaping raft about. We have bug reports. It's not good enough to have bug reports. This is something that.

There's a. There's a. There's a bug report. I mean, there's a pull request open with a fix that's in review.

All right. Okay, fine. That's what I wanted to hear. Okay, so. Okay, so that.

That's basically all the examples. So now we're basically caught up.

Okay. Okay. Oh, boy. So that was okay, right? And it's blindingly fast. Wonderful.

Yeah.

Okay, so now.

So first, okay, just for reference, there's a list of symbols. We don't need to go through it yet. So right here, here are the outstanding questions though. So the first one is, is null pointer queue. This is a trivial function that just takes opaque raw point. If you give it opaque raw pointer of zero, it returns true or raw pointer of zero, comma, whatever returns true, everything else returns false. It's a very useful function.

I think it should be null pointer Q. I don't see the point of having a roar in there.

So that's what it's called right now. It's called null pointerqueue. The argument for the raw is just that, you know, the functions that make them are the raw memory functions. We have to and from raw pointer in the compiler.

Yeah. Everything else is raw.

Well, okay.

Basically every symbol that includes the word pointer right now has it as raw pointer.

So this would be raw null pointer Q. Is that stupid raw?

No, well, no, but it breaks the consistency. Right, like, like the other ones are opaque raw pointer and raw pointer. Right. So those the raw and the pointer are never separated.

So I see, so you're saying null raw pointer Q. Yeah, all right, okay, I agree. No, raw pointer Q is more consistent.

Okay, fine. I think, yeah, I'm not sure which way to go, but I think that's fine.

Okay, Memory managed option, we discussed that, right?

Yeah. So the answer there is to just. Is just to. To not have the option. If you want a non memory managed one you call unmanaged object yourself.

Where's unmanaged object on this list?

It's not because it's already in the compiler, it's a compiler only thing for a similar purpose. For. Yeah, for a quite simple. It has the same semantics essentially for in the compiler, but it has no top level behavior right now.

And for, I guess I'm looking at the documentation right now currently it says unmanaged object is only available in compiled code.

Yeah, well, that was changed, right?

Yeah.

And it would have a return value in the case of a managed object.

Yeah. Which is almost identical to what it does in the compiler right now.

Exactly, yeah.

The only difference right now, I mean this is, this is one thing, it's a little annoying, but it's not. This is a more complicated thing that I don't think we'll be able to resolve right now is that the compiler managed type does not have the same payload value mechanism that the managed object does in the top level.

So in other words, with costs, what does it do? Does it have a different way of handling costs.

It basically doesn't have a mechanism for handling casts. You basically have to, you know, have just one copy of it. Like you'll have to unmanage one and then make a new version with a different type or whatever.

Right.

Okay, but, but yeah, all right, so, so the. Right. And we already talked about last time supporting unmanaged object. Unmanaged object for the managed objects in the top level. So, yeah, that's something we're gonna do anyway. Sorry, that sounds fine. Okay, so the last one is, get this, a function that needs a name. And you know, there's a question of whether it should exist at all, but I think it should. So this is basically. Right. How to describe this? Okay, so when you, when you. You've DL opened some dynamic library, basically when you. DLSIM is then the function for getting function pointers. Right? So like if I want to say, give me the address of this function you call dlsim. Right. So basically what DLSIM does is you give it the name of a symbol and it says, this is the address. Here's a pointer to where that symbol has been loaded in memory.

But by symbol here you mean symbol in the sense of a linker, not symbol in the sense of an expression.

Correct. Which is very confusing. But so the basic. Right, so the point is that that can be used for. Basically you can use that to get the address of a function pointer, but without loading it as a foreign function object. That's a quite marginal use. The thing that this is useful for is globals, because you can also have globals and dynamic libraries and this is how you load them.

Oh my God.

So you'd say the name, basically foreign pointer load. Right, you'd say foreign pointer load. You give it the library in the name and it would give you an opaque raw pointer, and then you'd use raw pointer to cast that to whatever it actually is.

Okay, but in this case, I mean, to me, I think it has to be name, not symbol. I mean, could say something like foreign name pointer.

Right, that's fine. Does it also work for functions?

Yes, they're names too.

So you can read the address of the function. I'm not saying you're.

Let's understand exactly what's in it. Ok, so you give it.

You can try that. There's an implementation. If you try that thing with the long name below, you can see what it looks like. Although there's so few error handling things. You can try the compiler demo base and then add one as an example.

How do I Do that compiler.

Compiler demo base the same as the example above. Right. Because functions are just globals in the.

And then I just do like.

Yeah, run it. You'll notice actually if you say, try running information of Add1, where Add1 is a symbol that you defined above. Oh. Or not. Oh, no, you just used percent. That's right. Okay, well.

What'S it going to show me?

It's gonna.

It's the same number. That's what. Yeah, it's gonna show.

I don't understand how it's got any number.

Oh, you mean. Yeah, I see.

Yeah. Yeah. It is the address of the function.

Yeah.

Okay, so the issue is somehow you got a spelunk. I mean, this is essentially a spelunction function, so to speak, that's gonna go into the symbol table.

Of.

The library.

So maybe for reference. So on Unix things, this is called dlsim for symbol. On Windows, I think this is called get process, address.

And yeah, I mean, I wonder also if. And I don't know if it's possible to implement, but if there shouldn't be a find function analog of this, where you can search the symbol table for symbols that match a string pattern.

As.

Opposed to just a string.

Yeah. I do have another question. How do I get a list of all symbols in a.

Does FFI provide that?

Libffi Definitely doesn't. They don't even provide this. This is just from the os and actually I'm getting this from some kernel functions which give you this. But I think in general, especially on Windows, I'm not sure you can do that, like even just get a list.

Of the entry points.

I think so.

I think you're right, Christopher. Yeah.

I think Unix might have a mechanism for it, but it's like, pretty bizarre. And there's this whole weird system for like. Right. Like, I'm not sure you can necessarily get the list of symbols associated with a particular library as much as you can get the list of symbols that are in your current process and you can prioritize, you know, a particular library, if that makes sense.

I mean, linkers know how to do this.

Yeah, sure. That's what their business is.

But they're. Yeah, yeah.

I think it's a funky business, though, and it works very differently on Windows, so.

Yeah. And I think, for what it's worth, I was just doing some Googling. It seems like there's definitely no posix function to do this. There's no standard way to do it.

Yeah, you're correct.

Right.

Yeah. And the answer is also going to be like. If you're on Linux, you could try manually parsing the ELF headers and presumably similar such manual parsing of object code on the other platforms as well.

Yeah, yeah. I mean there's three, there's three library formats, remember as well.

Okay, guys, guys, guys, guys.

Right.

What about something like named foreign object pointer.

So two things. One, I like foreign being the start because everything pretty much everything else here starts with foreign.

Okay.

Second, I'm not sure the named. I'm not sure named is a useful term. A lot of things are named, if that makes sense.

Yeah. But when you got to do something where it's otherwise it's symbol which is even more unbelievably confusing.

Yeah, symbol I think is a bit too confusing.

So maybe it's object instead of. But foreign object load. It's not really a load, is it?

It's not really an object though.

Especially neither an object nor is it loading.

Well, that's why for that was the argument for foreign pointer. Find.

Functions usually return a list though, don't they? I mean.

Yeah. And they also usually expect. Can take various kinds of patterns.

What's an alternative to find?

I mean it could be foreign pointer address.

That seems not useful.

Well, the pointer is the address, Right. Foreign pointer get or something. Or get foreign pointer. But I don't like the ordering of get.

First lookup maybe that's better.

That's not bad.

It could be like foreign library, you know, name lookup or something. So it's because it's an operation on a library ultimately.

What about foreign name lookup?

It sounds like it's going to look up, like if you have some weird Croatian name, it's going to look it up.

Yeah, we have to remember that this is. Well, it's in a library, right? It's always in a library.

Yeah, I mean foreign name library lookup.

I don't really like foreign, but none of the, none of these other things include the word library. It's all kind of implied. And also the thing we decided last time or two times ago was that, you know, there's the library load, library function load, etc. Brand which is for library link and that these are foreign X for the FFI system.

Fair enough.

I don't understand your point though, about the other things don't contain libraries. So this shouldn't either.

Well, I just mean. I mean first of all, you know, the library function load could be or. Sorry, foreign function load could be foreign library function load or something.

Exactly. Yeah. Yeah. It's implicit in foreign I think, right? I mean, foreign symbol lookup.

No, we can't use the word symbol. It's totally confused.

Yeah, I do think symbol is a bit too confusing. And also I'm not sure that Windows calls it a simple.

How about I like lookup, foreign name pointer lookup or foreign pointer.

What is the name? I. Yeah, I think foreign pointer lookup is better. The name. I just don't think adds anything.

Fine. Foreign pointer lookup. Sounds reasonable to me.

Yeah, I like that one.

Looking up a foreign. It's looking up a pointer in this.

Foreign place based upon a key.

Right?

Yeah.

Fine.

Sounds good.

And in some world of the future, maybe it could deal with a regex or something. But that's. We don't know how to do that yet.

We don't do that.

Nice.

Okay, that's basically it. These are just a few outstanding.

But you're planning to do this, right, Christopher? Like int. Regular things as well?

Yeah, I. That. That one is a little. I need to look into that. Yeah. The other ones here, like. Yeah, that. So, so this is basically something we talked about like two times ago, which is that instead of writing C and you should be able to, like, instead of writing, you know, raw pointer, colon, colon, bracket, C int, you should be able to write int star. But that requires a bit of parsing stuff. And I don't know if there's some complicated cases which it presumably just won't handle.

Does Chris Cooley have a parser that works against those things? Didn't he build that at some point?

And I know, I'm not sure if it does exactly what we need.

Right. Like, I'm not sure if it's. If it's, you know, if there's a way of calling it to say I only want to parse, you know, type signatures.

Well, I believe we had a type signature parser also wolframalpha at one point. Maybe we still do.

I mean, we have a link to clang APIs, which Chris's thing transitioned into using, so it can do whatever Clang can do in terms of parsing.

Although I remember looking at Clang like years ago.

And it's not trivial. Right.

And it's a mess doing it through Clang.

Massively not trivial.

I thought we did this. I don't know what's happening here. I thought we get rid of the.

Space, just instar space.

I mean, this is how we generate the link to llvm, which has about a thousand functions in it. We use PANG link for that, I.

Think, at least for the zeroth version of this, I would probably choose string patterns because we're not even supporting that many situations here. Yeah, I think the only minor concern, apart from the fact that it's, it's a bit of. It's a bit, you know, it's not totally trivial to do, but the only minor concern would be that people could be confused because they'll have, you know, their very. Some. Some type that only exists in their environment and they'll be like, why hasn't this. Why doesn't this work with a, you know, I see. With the shorthand, what's some. But even things like, you know, oh, in C99 there's a thing called div T, which is a struct containing 2 ints that gets returned by the div function, you know, but like, odds are we're not going to support everything like div T, so people are going to be like, oh, you know, why have you never heard of this type of.

Eventually we need an interpreter for C types and we'll get fancier about that.

Yeah, yeah, but it's not just. There are two points, there's one. Yeah, I agree. Interpreter is probably the right way to do it and it would return whatever's the right, you know, raw pointers and stuff like that. Right. Type specifiers. But. But separately there's the issue of like, if you go to foreign, foreign function loads documentation, you can see a list of the supported types and it's like, obviously it doesn't include div T.

You.

Have to use the docu tools. Okay, so there's the list. Right.

And what will you say if they just say I see. If they just say enter. Well, this is pretty string. I mean, this is pretty straightforward string patterns.

Yeah.

What about things like the array specifications in C?

That's where it's a little more annoying. It should probably support those two.

What are they when they're disembodied? When they don't have. Does that have a syntax when they don't have a name? Right. I mean, if I say int x bracket 20.

Yeah, I think you just say int bracket 20, I think. Although I vaguely remember this being some weird stuff where like.

Yeah, that's not quite the same thing. I mean, that's, that's not a. Yeah, I mean that's.

Well, isn't this one of these things about C types that they can't actually. Like they can't be disembodied because there's this weird winding rule. I know that's function pointer types, at least where the name is kind of a weirdly integral part of the declaration actually.

I just. Right, so the case of like the array like int bracket 20. So right now the way we represent an array like that is sort of funky. I mean it's basically. It's the way that libffi does it, which is that you basically treat it as like a struct with 20 fields which I guess has the same padding and stuff. I mean, this is just inherited from the phi, right?

All right, we should, we should wrap it up here.

I have one more thing I would like to bring up, which is about just a general thing about naming. We. We have foreign function blah, foreign function blah. But the raw memory stuff is all allocate raw memory, import raw memory. Why don't we switch those over to the noun verb style of naming instead.

Of verb now the raw memory import raw memory export.

Yes, they just group so nicely that way. You know, all the foreign function stuff is done this way.

So the only argument that the reason why it was done in that order was because the other import export things are in this section somewhat backward order, as are like binary read and binary write.

Binary write isn't in that order. Binary write is in the order with binary first and import raw bytes does a different thing.

Oh no, yes, you're right, you're right.

Import byte array. Well, import byte array is like import raw memory. But you could argue that raw memory import is importing from a different source.

Yeah, it's really not that much like import, is it?

No, I don't think it is.

Yeah, I mean it doesn't have the import export format, but it does have import in the name, so.

So I'm guessing. Okay, so what would be affected by this? I mean it will be raw memory import raw memory export raw memory read, raw memory write. Actually those are quite nice. Seems to me raw memory free.

Yes.

Right. Every one of them would just be flipped around to start with raw memory and they all group perfectly.

I don't, I don't hate that.

Yeah, I don't either, Christopher.

Yeah, I mean that's fine.

Okay, good catch.

So if we have create foreign callback, they're a compiler. There's like create data structure and create.

There's a ton of names in the system that are verb noun nowadays. They were mostly noun verb from in the old days. I just think that we should strive for new namings to stick to the noun verb style. I think it just.

What is foreign callback again?

That is. Let me see. Oh yeah, I may actually made an example for that. Actually if you go to the docs for create foreign callback, there's an over complicated example.

Create foreign callback.

Yes, right. So why isn't that foreign callback create.

Well, because it's trying to look like create data structure and create type instance and.

Yeah, but right, if it's parallel to other things, then yeah, I think it.

Has to be this way.

Okay, so, right, so basically here's. Here's an over complicated example that basically that code just needs to be in compiler demo base. But, but yeah, so basically there's a function that takes a function pointer. What Create foreign callback basically gives you a thing that can be interpreted as a function pointer, but which internally calls evaluated code.

I see, I see.

So see that takes pretty nice.

It's nice.

So look at the. Scroll down a teeny bit more and you can see there's the foreign callback that's just calling echo evaluation of, you know, hash squared. And it, you know, it's running in your kernel called from C code.

It's pretty nice, right?

I mean maybe it should have a more, a more magnificent name.

I mean this is the name we came up with last time there was much discussion about it.

Okay, fine. From, from a piece of C code, but now I see, I see, I see the way it has to work is you define a piece of C code and if you wanted the C code to be in charge calling main, then it's a different pattern than in this case where the, where the Wolfram language system is in charge, loading a library which contains C code which then calls back into Wolfram language. Right. In other words, if you wanted something where fundamentally you execute from the command line through the main.

Yeah, right. Then this does not help you.

Right. How do you do that? What is the best modern way of doing that?

Well, you're describing a bit of the standalone application stuff, I think.

I see where you have a main program that can call into this.

I'm not sure if it's calling into this exact name. It's calling into the kernel in general. But.

Yeah, right.

But I think this is useful for stuff like. Right, like a few examples. Like one example would be something like in SQLite or in Postgres they have systems for like, if you have. Oh, what is it? It's like if you have your own, you can have like weird functions that are written in the query, but which call out to arbitrary C code. So here they can call it to a kernel.

Yes. No, that's great. Right? That's great for postgres, for example, with some weird, you know, compute the bessel function of this number. Right. Excellent stuff. All right. Looks like a lovely system. All right. Anything else for right now?

I don't think so. Otherwise, I think this is probably wrapped. Wrapped, yeah.

Good.

Okay. Excellent work. All right. Okay, for people on the live stream, we are about to go to a different live stream about calcus and algebra. See you all there. Bye for now.

Bye.