Calling External Libraries with the Compiler

In the title, I'm going to be talking about connecting to libraries with the compiler. So first, before I get into exactly how all this works, just a little bit of sort of motivation. So basically with the stuff I'm going to show today, the main advantage is that sort of you can make these links that are very, very efficient. So in particular sort of as efficient as if it was written in like the, the core C kernel of Wolfram language. So that means that there isn't any data transfer costs and things like that. So for instance, if you use WSTP or Library Link, which are the main existing ways of connecting to external libraries, there's often data transfer costs where it has to either do copying or you have to do copying and things like that. And this can cause overhead and make things slower and so on. Another big advantage is that there's no expression serialization. So that is again, if you're using WSTP or Library Link often, if you're passing just a general expression over a link, the expression has to be serialized as WSTP and then you have to read that serialized version of it, which is often very painful, and then you have to process it and return a serialized version. And this whole time, if you have some giant gigabyte sized expression, this is very slow. Apart from the fact it's also pretty inconvenient to sort of manipulate these serialized expressions. So with this setup, you don't have to do that. You just sort of directly get access to the raw expr, which you can manipulate using much higher level, more Wolfram language style functions. The last advantage which I find very convenient is that everything happens within the kernel and within sort of, you can do everything within a single sort of session. So again with WSTP and Library Link, WSTP and Library Link, you need to create these interface files, you often need to compile some C C program, make some dynamic library and then link it and so on. With this you can do everything just in a notebook in one session. You can also do things where you do them in packages and cache the results and so on. But it's possible to do them in just one session, which is extremely useful for development, among other things. So let's get started with maybe a very simple example. So this is the signature for a function from OpenSSL. OpenSSL is. It's a library that it basically does cryptography and it's used like it's used in Mathematica for like the encrypt functions and hash and stuff like that. A lot of that Uses Open ssl. So this is a function I just got off the documentation, the online documentation for Open ssl. It's not a very interesting function, but it's a very simple one. It just returns the version number. So I'm going to try and write a little link to this function so that we can, you know, get the version number of our open SSL library. And basically the main thing we're going to be using here is this new in 13.1 symbol called Library function declaration. So let me show you how that works. So basically what I'm going to do is for every external library function that I want to access, I'm going to create a library function declaration that basically tells the compiler, what is this function called, where can I find it? And what is its type signature? So we'll write one for this, this function here. So the first argument to library function declaration is the name of the function that we, that we want to be able to call. So in that case, it's just this open SSL version num thing. The next argument is normally the path to the library where we can find this. So it would be, you know, some, you know, lib, crypto.dylib or something like that in the case of open SSL. In this case, though, OpenSSL is actually always linked whenever you start a Mathematica session. Because, because it's used in some sort of core areas. So I can actually write automatic here. So that's what I'm going to write for right now, so I don't have to write the whole path to the library. And then finally, you need the type signature, and you write the type signature in the format of the compiler type system. So, so the compile type system is all documented, but I don't want to get too far into it. But essentially in this case of this function, you see, it takes zero arguments and then it returns an unsigned long. So I'll just write that as C unsigned long. So it's an unsigned long in the style of C. So, you know, Right, so if I run this, this will be, sorry, I have to load the compiler the first time I run it. But you see this, this is a totally inert object. It didn't actually do anything here. This is just an inert thing. If I actually want to use this, just like all the other compiled declarations, I have to feed it to the compiler. So the easiest way to do that is using function compile. So what I'm going to do is function compile and its first argument will take A set of declarations. So in this case I'm just going to give it that library function declaration, and in the second argument it takes the program you'd like to compile. So in this case I'm going to have a program that doesn't take any arguments. And what I'd like to do is I'd like to call this function here. And because I'm including this declaration in my call to the compiler, it means that now there's a function that I can use in my program that will call out to this library. So by default, the function that it defines will be called library function of, and then the name of the function from the external library. So this thing here represents the library that we declared with the, with this library function declaration. So that's the function. And we see it takes no arguments. So I'll just, you know, apply it to no arguments right there. Um, so I'll, I'll run this and let it, let it compile. So essentially all this is going to do. It's going to compile this function and all this function does there's going to compile this program, and all the program does is it calls this function with no arguments. So you see, we get this compiled code function back, and if I now call that again with no arguments, we get back some number which represents the version number of open ssl. So it's not a super exciting example. You can see it did, it did work. So this is just. This is, I think, kind of cool though, because the amount of code needed to write a link to this very, you know, admittedly very simple function is very short. We didn't have to open up a text editor and make a C file and run GCC or M prep or anything like that. It was just a couple lines in the language, which is, which is very convenient. So let's move on to more real or more useful function. So this is another open SSL example. This one is the SHA256 function. So this is a hashing function. There's a version of it that's actually built into the language through the function hash, but we're going to make a link to the open SSL one anyway. So this is the signature, the SHA 256 function from open SSL, which you can see is a little more complicated. And basically the way it works is you give it essentially an array of bytes and then you give it the length of that array. Because in C, of course, you have to tell it how long your arrays are. It doesn't know. And then finally you give it a buffer into which it will write the results. So you have to allocate an empty buffer and then feed it to this function and it will write the results in, and then you have to read the results from the buffer. And, and I think that the return type is just, it just returns the buffer that you gave it. So the return type is not very useful. So let's just like before, we'll make a library function declaration for this and it'll be basically the same idea. We'll write the name of the function, the fact that the library is already linked, and then for the type signature, we'll see. It takes a few arguments. The first argument it takes is, well, I'm actually going to write this as a C string, which is just sort of a type that we have that sort of represents, you know, like an unsigned char star could be a string. This is. But I'm going to make it so that this function that we're going to compile takes a string and hashes it. So this would be a convenient way to do it. So it takes a C string in its first argument, then it takes a C sizet, which is this count argument here. And the final argument it takes is basically an array of bytes. And so the way I'll write that is it's a C array of C chars. And this is a new piece of Syntax as of 13.1, so you may not be familiar with it yet, but basically this represents a type specifier, which is basically a way of writing parameterized types. So in this case, Carray is not really a type in itself and needs to know what it's an array of. So in this case it's an array of chars, where char is just the C byte type, essentially. Right, so this is. Oh, sorry. And then we need the return type, even though we're probably not going to use it. And it just returns the same thing you give it in its third argument. So another C array of C chars. So that's our library function declaration. And now we can try, now we can try compiling the function that calls this. So just like before, we're going to make, we're going to call function compile and we're going to give it this declaration and then a program. This time the program is actually going to take an argument. So we're going to take, say, a string that we're going to hash. And now we need the body. And I know I'm going to need some local variables. So I'm going to make a module. So the first thing we need to do is let's try and get this C string, because right now we have a Mathematica string, but we don't have a C string which is just a raw array of bytes. So do that we can use a new as of 13.1 sort of compiler only function called cast. So there's a few compiler only functions that we're going to see here. These are basically functions that usually have to do with things with the type system and and so on, where they don't necessarily have a real meaning in top level code, but they're important for making compiled code work. Because again, for instance, in the compiler there's a type system, everything is typed, but in the top level there is no such type system. So it doesn't make sense necessarily to cast between say a 32 bit integer and a 64 bit integer, because in the top level those are the same thing. So in this case what we're going to do is we're going to cast our string that we had input to a managed C string. So this managed here, maybe at the end I'll have time to get into it a bit more. But essentially the managed, the managed type is a way of wrapping these sort of raw unmanaged types with memory management. So normally if you just had a C string, there's, you know, you have to allocate that and then when you're done with it you have to free the memory. But of course it's much more convenient if it's reference counted and you don't have to worry about it. So basically the managed type wrapper allows there to be sort of automatic memory management for things and you can define that behavior yourself and so on. But there's an automatic version for C strings, so we'll call this the plain text because it's the thing that we'd like to hash. The next thing we need is we need that buffer into which we're going to write the output. So to do that we're going to use another sort of one of these compiler only functions called Create Type Instance. So this lets you, if you have a type, it lets you create an instance of that type. So in this case we want to create an instance of this cra of C chars. Actually we're going to create a managed instance of a C array of C chars, which is sort of a lot of nesting, but essentially it's just, we're making it, we're allocating an array and we want it to be memory managed. And you can look up, you know, if you. All this is documented, everything I'm showing here is documented. So if you look up, for example, the documentation for C array, it'll document this usage of Create type instance, this constructor for C arrays, and it'll say that you can give in the sort of second argument of Create type instance the length of the array that you want. So in this case, I happen to know, because I've read the documentation for open SSL, that this buffer has to be 32 bytes long, which makes sense because it's writing a 256 bit hash, and 256 over 8 is 32 bytes. So that we'll call this the ciphertext because it's the, it's the, it's the buffer into which the ciphertext will be, will be written. So now that we have that, we can actually make the call to the library because we have all the, we have all the inputs we need. So just like before, we'll write library function of SHA256. This represents the function that we declared above. And then we're going to apply that to some arguments. So in this case we'll apply it to the plaintext, then we'll apply it to the length of that plaintext. So I'm just going to write string length of structure. There are some instances with UTF8 where I think this is technically a slightly unsafe thing to write, but I'm going to ignore it for now, and then we'll write the ciphertext. There's actually one other detail though, which is that, well, string length returns some kind of integer. I think it returns a machine integer, which is some signed integer 64 or something like that. But here we require that it actually be a C size T, which is a particular kind of integer. And so again, this is something that you don't have to worry about in the top level, where all integers are kind of the same, you know, from, from the outside. But because the compiler is a type system, we actually have to tell it, we have to cast it to the right kind of integer. So here C size T. So once that runs, that will populate this, this buffer, the ciphertext buffer. And, but, but now we actually have to read from that buffer and turn it into something that we can return to the top level, so we can actually, you know, use it outside the compiler. And so for that I'm going to use this create type instance thing again. And this time we're going to create an instance of the type numeric array and, and again this is all documented stuff. There's a usage of numeric array where you can give it a, a C array and then the length of that array and it will make a numeric array out of that. So in this case we'll give it the ciphertext C array and we'll tell it its length 32. So I'm going to start that compiling now. But again just to recap, I mean all this is doing is we're taking a Mathematica string, we're casting it to a C string, this sort of raw raw format, then we're allocating a buffer into which we're going to write the output, we're calling the function and then we're wrapping that output in a nice numeric array so we can return it to the top level. And so now if I, if I take this SHA256 function and we'll try giving it a string, you see it returns this numeric array and we can maybe run normal on this to see what it looks like. But we have a plausible list of bite sized numbers so that, that's something we can maybe check that this is, this is legit. So maybe compare it to the built in hash function. I think I can just do this. Ok, we get the same answers. That's a good sign. Another nice thing is if we look at, it's very very fast. So if we look at for example the built in hash function, I'll use repeated timing and you see it takes 10 to the minus 4, 10 to the minus 3 seconds, something like that. But if we look at this compiled version that we just made, it takes like 10 to the minus 7 seconds. This is like fours of magnitude faster than even the built in one. And so I mean this is just to say that this is like extremely low overhead, it's very efficient and so on. So you can really make really low level efficient links this way. So maybe at this point I just have a pause to see if there are any questions about this. Yeah Ian, what's the relationship? Type hint. That's a very good question. So there's a, actually this is a new 13.2 symbol called type hint. And basically. Right, so basically what the difference is is if I write a literal like I write 256 in my compiled code, it will sort of try and figure out what's the type of this number 256 and it will say oh, it's probably a machine integer or something or maybe if I add it to Something that's definitely not an integer, it'll say, oh, maybe that was a machine. Real actually. But there are some cases where it struggles to figure out what the type is. And if you want to help it out, you can say type hint. Oh, this is actually a complex. This is actually a complex 64 or something like that. So type hint is really for telling the type system when it's instantiating something. If it has many options of different types to choose from, it helps it pick which type. Cast, on the other hand, does something slightly different. It has something where it knows what this STR thing is. It knows that's a Mathematica string, it's definitely not a C string, but this will actually do the conversion. So type hint generally doesn't do any conversion, it just tells the type system which choice to make. CAST actually does conversion. So, yeah, maybe there's one other question. Yeah, Sure. So there's maybe two different interpretations when you say data structure. So one of them is that the literal data structure thing in Mathematica, these are totally supported, these are implemented in the compiler and you can pass them to and from the compiler in all sorts of nice ways. If you want to talk about other sort of bigger expressions and things like that. That is exactly what I'm going to get to in my next slide. So hopefully I'll address that. Let's see here. Right, so, okay, so this takes a little introduction. So basically there's a geometry library called geos, which basically supports very efficient Euclidean geometry operations. And I'm not going to show, I'm going to sort of glaze over some parts of this because the code is. Can be kind of long to just, just handle all the cases. But basically I'll show you how to like, given that I've already written a lot of the basic stuff, how you then, you know, add another function to something like geos. So let me explain. So if I run needs geoslink, essentially what this does is it loads this compiled component it's called, which is sort of a new 13.2 way of encapsulating compiler functionality, which, if you want to learn more about, you should go to Tom's presentation tomorrow morning. But basically what this does is it introduces a bunch of compiled symbols. So for example, here's a function called togeos, which will take, say, some geometry stuff. Ignore this message. That message is just a bug. But right, so here I'm taking this polygon object, I'm passing it, I'm basically using this compiled symbol and I'm turning it into this GEOS geometry object. And now I can call from GEOS on that GEOS geometry object and get back the polygon I put in. And so. Right, so I'm. I've already written that code that basically does the conversion and it works in exactly the same way as what we just saw. It's just you pass in an expression, it uses compiled primitives to rip apart that expression and then it calls all the right, you know, library functions from GEOS to then construct the appropriate polygon, line point, whatever in GEOs. And then we have this nice mechanism with these managed expressions, managed objects that can be returned to the top level reference counted and so on, and managed very nicely. So I'm just going to. I've very little time left, I think, so I'm just going to kind of blow through this example a bit. So this is another function from GEOS that maybe we want to support. So in this case it's a function that basically you give it some geometry and it sort of like puffs it up. It's a bit like region dilation. So let's just write a connection to it. So here I have a couple of declarations that are required for the. Just managing the. Just managing basically the managed geometry objects that get passed back and forth. But basically we're going to write a library function declaration for. For this GEOS buffer thing. It's called GEOS buffer. I think we can. We can. The library is already going to be loaded and then the type. It's going to take a GEOS geometry object, which is something I defined in that typedex area. And then it takes a double and an integer and it returns a new GEOS geometry object. And so the function that we'd like to actually Whoops, the function that we'd like to actually compile, it's going to take a few arguments. So it's going to take a. It's going to take the geometry in the form of a managed GEOS geometry object. And then it's going to take. See, it's going to take this, this width argument, so width, and that's going to be this C double. And then it's going to take this quad segs, which I think tells it sort of the resolution at which to buffer. And that's going to be a C integer. And then for the actual implementation of this whole thing, it's actually going to be pretty simple. So we're going to call library function. We're going to say GEOS buffer. We're going to pass it all of our arguments and then we're going to take the output and wrap it in one of these managed objects. So again I'm going to say create type instance and I'm going to create this managed object. And again in these typedex thing that you can come and ask me about it later if you want to know more. I include basically the information it needs in order to know how to memory manage these objects. So if I run this whole thing I should, whoops, I'm going to give it a name. We'll call it buffer maybe. Then what I can do is I can get some polygon, say and there's some random polygon. We can, we can visualize it and we can try running this operation on it. So we'll say to geos, we'll give it the polygon, then we'll say buffer and we'll give it that managed geometry object. Maybe we'll buffer it by 0.1 units or something like that at a resolution of 5, it gives us a new GEOS geometry object and we can say from GEOS of that and get back this buffered polygon and maybe we can visualize these together. So we'll make the, the red buffered polygon and then the blue original polygon and there you go, you can see this sort of puffed up version of the polygon. So. Right. And again, just like before, this is all nice and very efficient and almost all the time that this spends running it spends inside of GEOs as opposed to, you know, as opposed to out, you know, doing, doing links and transferring data and so on. So you can send like big, like megabyte sized, you know, million vertex things and it's very, very efficient. So yeah, so if you want to learn more, I'll be probably around somewhere outside or you know, come talk to me and otherwise you should go to Tom's talk tomorrow morning to learn more about compiled components and some of the other compiler stuff. So yeah, that's it for me.