The Future of API Integration

So, hi, good morning, my name is Giulia Sandrini and I'm the lead developer in the ML team. And today. Well, today I'm not going to talk about ML at all or tangentially because what I'm going to present here is our advancement in API integration and what is going to come partly in version 14 and partly after 14.1, or how would be called the next version in terms of API integration. So why is it interesting why I'm doing this as a ML developer? Well, okay, short answer is large language models, but slightly longer answer is that lot of technology has been moving towards server distribution. So this is very interesting for us and I hope it'll be interesting for you as well. Okay, so what are we gonna look at today? Very, very quickly, what is an API and why it is useful? I'm gonna give some superficial technical details, show you how you can define one using Wolfram language, and then move to the meat of a talk, which is our current service Connection framework, our abstraction layer on top of API connections, and sneak peek into the future, like what the new service connection framework is going to be and how it interacts with our company wide LLM effort. Okay, so what is an API? API is Application Programming Interface. And, and basically you could see it as a system, a set of rules that allows different components, different software, software, hardware, people and software to communicate. The idea here is that you define a clear boundary of your application and the gates in this boundary are the API endpoint, the access ways to your application. So one typical example is imagine that you're in a restaurant and in our example the API is a waiter and you as a user or customer, you don't go into the kitchen and you don't need to know how the kitchen works. You just talk to a waiter, you have a menu that clearly explains what you can get. You tell the waiter what you want and it's the waiter that goes into the kitchen and explains it in the right way and decides how to assign things and basically does everything completely hidden from your site and then comes back with your dish. So I put here some question marks because you don't know and you don't need to know what happens in this interaction. This is internal details of a system. So using an API structure makes it very clean to the outside user to communicate with your system. Now let's see, I need to scroll to inside here. Okay, so why do you want to use APIs instead of not using them? So first thing is you can make your system very modular. So for example, you can have like diverse application talking together. Imagine you're on a phone. The phone has like GPS as a camera and the operating System has some APIs that, you know, phone apps can talk to and they can use the camera and they can access the GPS without needing to know the specific of the hardware. So this is much, much, much simpler. And you add a new piece of hardware, you have a new, new API that you can connect to. So it's all very easy, it is efficient. So you don't need to redo things from scratch. You can just build upon an existing API to create new things. It can be used to automate a lot of routine things and expand on top of services. Imagine that your company offers, I don't know, maps and then users can build things on top of your maps using the API that you provide, which is something like OpenStreetMap, for example, security, my metaphor of the boundary, the gate. You have clear access points so you can put a password, some more sophisticated authentication methods. And you control exactly who can come in and what do they have access to. It is scalable. Like you can add new endpoints, we're going to see what they are in a second. So you can add new functionalities basically to your application without impacting the older one and without restructuring everything, at least from the outside. And last but not least, you can put a price tag on API access so you can make money out of that. So you can see several good reasons why you probably are more I just the few initial ones that could come to mind. Okay, so let's see some kind of like practical, practical cases so that it maybe feels a bit abstract so far. So you might have encountered APIs in your daily life, especially if you have a phone. Imagine that you have a weather app. The weather app. What it does is it connects to a server and asks for, based on your location, it asks for data. The API answers with a table or a JSON or formatted data of some form. But you don't see that. You see the picture of a Sun or stuff like that. So that's a functionality built entirely upon the data provided with an API. The APIs are used to do social media sharing. You want to, you know, if you read an article and you click share on Facebook or you see a picture, you see share on Instagram that is using the Meta API, for example. Now the typical one is location based services, like all apps that offer deliveries or car sharing or public transport information. They, they connect, they connect to local APIs from the different administrations. They use the API from your phone to get your position and send it to one of these companies. So there's a lot of APIs here offering you together that are working in concert to offer you a service. Okay, so let's get to code. This is all very cool, but can you do that in the Wolfram language? Of course, we have a system of instant API, as they're called. So let me show you here, this is the core of it. It's called API function. This is using the Wolfram Cloud. And basically here I'm just saying that this API will take one parameter, which is country. And I'm here using interpreter syntax. This is like this is interpreter type country, so it would be validated. And the function that the API runs is get the country and tell me the population of the country so I can create the function and I can deploy it. So now this is the name of my API and this is the so called endpoint. So this API has one endpoint which is called population. I can do a request to this endpoint, again, execute this URL, appending a value, say country Italy, which is where I'm from, and I get an answer out. So basically as an external user you just do this. You don't need to know how this works, you don't need to know about databases, you don't need to know about anything. You just need the working Internet connection. And then let's say you can extend it. Let's say that my user asked me, oh, it's cool that I can get how many people are in Italy now, but I want to know how many people were in Italy in the 70s or 80s. So I slightly modify my definition as a country and year. Year is an integer and is converted to a data object and I extract the year and. No, sorry, this is the default. The default is the current year. My mistake. And what I do, I do a slightly different entity. Value call asks for a dated property, not the population, but the population in that year. I deploy it on the same URL and now if I pass two parameters, I get how many people were in the 70s. So you see here the code change might have been massive, but the way to call it is actually very, very similar. It's just one extra parameter that the user needs to specify. So this is hiding all the complexity. And we want to hide this complexity as well. How does API calling work in the Wolfram language? Well, we have a system called. Oh wait, so I have an introductory slide. Sorry, I forgot. So why do we want something better? Why do you want to have started. Well, this API that I just defined is pretty simple. It has only one endpoint, population, and in the second version it has two parameters, country and year. So I can imagine that you keep working, you get more feature requests, you add several different endpoints, not just population, but areas, regions. You have an old rich country information database and you want to make it all accessible to the users through a well structured API. So you have population by sub region or province or municipality and all things like that. So then building all these URLs by hand and passing the parameters by hand, and sometimes they are strings, sometimes they are numbers and they have to be converted and things like that, it can get very cumbersome as a user, maybe you want to use several APIs. So we provide a way to keep it organized through what is called the Service Connection framework. So the Service Connection framework is not so recent as being added, I think around 2006 earlier. So that was not of course done in Machine Learning Team. It's just system connectivity. And the idea is let's integrate third party APIs into the workflow language. So what are the core ideas of the frameworks? Well, you have pre built collections, so you don't need to know any URL, you don't need to know things by heart. So we have a list of things that are maintained. Unfortunately this year a lot of people deactivated the free APIs, so some of these will have to go. But these things can provide instant access to all these services. Authentication. So I told you earlier about the fact that you can use API to control access. So many of these you need to authenticate or authorize, provide information about who you are. So all of these can be cumbersome. You need to know how to pass API keys, you need to know how to use OAuth or there are several complex systems that can be set. So the Service Connection Framework is dealing with that for you. I have here a few examples. For example, the Archive API is an open one. There's no authentication. If you do service connect archive, then you just get this pop up, you just have to agree to the term of service and optionally you can click this button. This will save your choice. In this case, no credentials, but securely in the cloud. It's tied to your cloud account and then you're good to go. Other APIs like Federal Reserve, Economic Data, this requires a key that you have to generate. So the way to start using this is just you go to the website, you click on this link, you generate or you get your own Key, you put it in and you connect. And then here you see that it's a bit more useful to save the details so you don't have to do it every time. And then there are schemes like OAuth, for example, Twitter is like that. So this is from our Twitter connection and this is a screenshot from my browser. So in here you have to authorize the Wolfram system to basically do things on your behalf. So you can like retweets, post tweets, stuff like that. So all of this is non trivial if you want to read by hand and is a big part of the reason why we do have the framework now what do you do once we have these services? You can authenticate. The idea is that you can do requests like I did before and many of these systems we had that instant API that was returning the population of a country as a quantity object. The typical APIs will return things like as JSON strings or stuff like that. And there's some overhead to parse that. So here I do this service connection. I'm already connected here. So if I want to show you. Well I'm not going to show you live. Let's see what happens here. If I do this, I get a pop up, it's on my other screen. So I get this. If I save, I don't have to do it anymore. Agree and I create a new connection and then I can do things like for example here I search for Quantum eraser max item 4s and I want all the articles when they are published in the title. So yeah, I can do this live here I did it twice. So this tells me that I had two connections. But you see this is quite easy. I want more of them. I do this. This is a data set. So this is not a string that you have to parse. You don't need to know what's the specific output type of this API. So that's another good reason why you want to use the framework. So the output is conform and the input is conform, like the way you pass the parameters. In general you pass only strings, but yeah, I'm passing the number five. In some cases you might pass an image or things like that. Let's say that you want video subtitles. It would be cool to work with a video object that we have embedded. So all these things are taken care for you by this framework. So why I'm talking about this framework now because this framework has a few pain points. It's quite old and hasn't been touched in a while. And now that we've been stressing it a lot this year as users. We decided to make it better and I think there's going to be great value for everybody. So that's my main point about talking about this is how we want to make it better. I think I have one more example here. Oh yeah, here it is. This is what I was talking about before. You get stuff directly in mathematical formats. So for example, this one here is giving. I was showing before federal server chromic data, this is giving me a time series. So I get this output and bam. I can just keep working as plotting the movie median and using datalist plot in getting automatic labeling and stuff like that. The usual stuff that you can do in the Wolfram language without even stopping to think that this is a call to an external service and. And you have some friction in getting the data in and out. So yeah, I mean good example about why you need it. Okay, so I mentioned few pain points. What are the main ones? Well first what we encounter documentation like each one of these connection is documented and if I go here and I do a search for this, let me put it in from my other screen, we get this out so you can see what requests you can do. So this is very cool that each one of these has all the elements. But this is about the specific API service. The framework itself is a bit lacking on this regard. So what if you want to add new one? Can you? Well, you can, but it's not clear how. So it's one of the reasons why we want to touch this to make it very clear, very open, very streamlined authentication. I showed you several ways to authenticate but would be very nice given that we are programming language if you could do these things programmatically, if you could do this thing with other credential storages other than the Wolfram Cloud, if you want to build an application on top of this. There are a few things that are missing here and extensibility. So I don't know if ever an external user tried to create a new service connection. It is quite complicated. So we spent the last couple of months trying to make it simpler, get rid of boilerplate codes and remove this complication. So these are kind of the reason why I'm talking about this now. And I want to show you how we can make it better. And I'm of course super open to feedback. So you should have pre release versions. We can talk about what's there and what's not and point about the future. So this is supposed to be a very open System, So no secrets there. Okay. Now I hope I didn't go too fast, but I want to reach the meat as soon as possible. So what is in the works for the framework? First thing, a clear boundary. So the framework itself has an API. You can manage the API, but you yourself are an API. So the framework has entry points and we have service objects. They are not truly stateless now. And the way you define the connection is absolutely non stateless. It's only packet based. It's quite complicated. So we want to throw that all away and redo it from scratch. Adding a few convenience things. You might have different keys to access, e.g. chatGPT from your personal key, work key, your other work key. So all this is quite complicated. So we want to make these objects easy to find, easy to remove, easy to edit. The typical things that you expect to do in a highly symbolic programming language. The main function that is used both under the hood or explicitly is service execute. So this is going to, you know, the main go to function when you want to perform requests. I mentioned we don't support programmatic authentication. We absolutely want to add that asynchronous request. I haven't shown you this, but to give you an LLM based example, you've all played or many of you have played with ChatGPT, you don't wait for a few seconds and then you see the whole blob of code coming. You see words appearing one by one. These are achieved through server side events and right now the service framework is not built to support them. While the underlying URL functions like URL submit, they do support it. So that's, I think that oversight and we are going to fix that so you don't have to leave the framework to perform something asynchronous. That makes sense. And then why not giving you more information about what happened. We have underused symbolic payloads that we want to make first class citizen of the framework so you could get loads more information about what happened. Server errors and RAW responses from the server, raw requests sent to the server. So yeah, I mean lot of things, lot of goodies that we can expose here. Now I talk about authentication right now is there's some hidden ways to pass keys that I'm going to show you later, but is not clean at all. So we want to expand it using system credential so you can handle seamlessly and securely authentication via key like password like or something more advanced like oauth and you can store it securely in your computer. We still of course are going to have the authentication stored in Wolfram Cloud if you want, so you're tied into a specific machine. But there are also more things here that we have in the planning, but these two are the two key ways that you think we will be offering. And last but not least, extensibility. The whole idea here is that this is a flexible framework that can be extended. You can add new services all the time as soon as you need them. So what do you need to do that? Well, right now it's a lot of code you need. The framework handles these things for you for the built in connections. But if you want to add a new one, you need to specify all those things yourself. And that really makes no sense. So we've been working hard on providing seamless parameter checking like functions to retrieve responses, parse errors, report back to users. And all of these you don't even need to know because the idea is that this thing will work under the hood. So you just input your connection details and you define the schema of the structure of the request and that thing will work. So this sounds a bit abstract, so let's see some practical example of this. Okay, so here, this is a function that will come soonish, I think like in a couple of days to the function repository. And you know, this is not a framework design, but is like a sneak peek, like a preview. So this is basically transpiling new syntax into old framework. And considering that before you had to create a packlet with few different files and know a bunch of like internal esoteric incantations, I feel that now this is the level of complexity that we want to achieve here. I found this nice free API which is called Cat Facts because of course, cats. And here I'm just specifying a service name URL and what we still call internally row requests. That means something that is not going to behave like a warfare language. Functions with quantities and things, just basic type validation and sending stuff back and forth. And I'm just adding one here, Cat Facts, of course. So I had this packet already, so I'm using this override target. True, but of course not needed. So this service here is loaded, ready to be used. And I think it was in my list of services before. And I can do service connect here. So this is. I saved my connection before this one without any key and now I can get a fact about the cat. Bam. Okay, I didn't know that. Cool, let's get another one. All the spirit of cats. Okay, I mean we can stay here forever, but you can see that this is the Idea we go from a very complex system to something that you can do in of course, a one liner. So that's the end goal here. Yeah, basically. So this is a key feature of vertical design. And now let me add a bit of. Why. Why am I talking about this from a machine learning perspective? Well, because we are building a lot of our stack on top of external services and we don't want to depend on anyone in particular. We don't want you to depend on our choices. We want this to be super flexible. So. I forgot the slide here. So this is like a sneak, sneak peek about something that is not even existing and what this will be in the future. Like we like here. I still have to do some massaging. So this will work like as soon as I publish it. So you can use it and you can get your own connection in. But this has to create a packlet. Of course you can distribute this on the packet repo. It's not bad per se, but it's still quite cumbersome compared to absolutely zero friction on doing everything in memory. So this is completely symbolic service connection that we had in the plan, but is not even here yet. So it's just the end point of this journey. And then now why, I was saying, why do we care about this? Well, we want to make our LM stack completely extensible. So like here for example, something which is currently not supported out of the box is using Hugging Face, the inference API of Hugging Face to use chat notebooks and things like that. So how can I do that? Well, I wrote, I mean this is quite a big connection, a lot of endpoints. So I'm not going to show you here. The talk is a bit beside the point, but I wrote a Hugging Face connection using the system like detailed above. And here is. This is the unfortunately ugly hidden syntax that you have to use right now. This is going to go away, but you know. Oh yeah, I was already connected. Well, let me do something crazy here, like a bit more Steven. Like I'm gonna quit, gonna quit and see. This should work out of the box. Here it is. So I don't want to show my key live. So I'm using the system credential here and I'm using the new. We have an abstraction of the abstraction layer. So we package all these services under something called LM Services that every LM function is using in the, in the system. So here what I'm doing is I'm using Hugging Face as a virtual, like a service for this virtual endpoint which is called Chat, which is then used by system functions. And this stuff is going to be documented at one point when we stabilize the design, its work is a. Is still in context because it's not finalized. Okay, so this is what I need. Basically once I have my connection, I just need to register it. And now the only thing, this is standard system function. I put something in here and in my evaluator I say use hugging face. Use this dialogpt large from Microsoft, which is the model I put as default here. And now it's connecting hugging face. Whoa, done. So this is why we were interested in APIs this year. So we want our whole LLM stack to be completely independent of our choices. Imagine that you have a local endpoint on your computer. So let's say that there are C libraries to run, quantize. So I'm also answering the first question here. You can set up a local API local endpoint in your local host, running quantized models as fast as possible. If you have a decent machine and you can connect to them using this. Imagine your company or your customer, they have their own API server. Like they have an instance on Azure or AWS or whatever. It's called the Google Cloud these days. Then you can just write a service connection and this is why we made it very easy. And you can enable our functions with that. So this is now completely flexible and you can use whatever. Okay, so this is the bulk of it, so thank you. I didn't have so much time. As I said, I hope I have not gone too fast. If you have any questions, I'll be around and you can ask me. Now. Let me put the camera here. Hello again. So, well, yeah, let me see for rest. API services are well documented. It would be very cool if a service definition could be generated from the documentation. This is a very nice idea. I didn't have time to work on that at all, but. But yeah, definitely something to look into. And let me see, the other one was. Yeah, okay, Bob, you also answered already directly that. Thank you. Okay, updates. So what is going to come? Let me detail what's going to be in 14.0 and what's going to be in what's after 14.0. So service registration. This is in Fortune 0. This is what our stack is built upon. And right now this is powering Chart Chat, Evaluate LLM, Synthesize LLM functions like all those LLM things and chart notebooks. So we have a couple of virtual endpoints here. Chart and chart submit to do synchronous and asynchronous chart request. I have him. You know, I didn't have so much time to work on this, but this is all packed updatable and I want to add few more endpoints for like dealing with images, image generations. So I want to expand from OpenAI to add like stable diffusion or stability AI, how do they call it? And few more APIs as soon as they come out. There's more LLMs like Aleph, Alpha, Luminous. Several things that are out there that we would like to connect to. Registration is there to stay and you can already use it. Custom connection extension in the language one is not going to be there in 14, but this thing here is. Well, let me see. Okay, so this is my. You know, I'm finalizing the documentation here. So here you will get. This is my private cloud, but it's sort of public. If you, if you screenshot this, this is going to change. So don't screenshot this. No, I'm going to add examples for all these. So how we can do this whole process requests and why you want to do that. This is an extended version of my CAT thing and here for example, I show you how you can do your custom connection to OpenAI to get embeddings out and how you can use. For example here I'm taking these embeddings and I'm extracting them from the row response and wrapping them in the make array. So instead of getting something like this out, I get something like this out. So you can build something that really like the regular framework. It looks like a Wolfram language function and it's going to be very easy. So this thing here I won't release in the next few days, so you don't need to wait for next version. Other things though, like completely stateless definitions, hot reloading, if you want to play with those, hot reloading of service connections, those things require deeper framework redesign and they're going to come in the next version. So roughly this is the split. Ok, let's see if you have new things. Okay, well, we don't have. Oh, thank you, Amy. We don't have any more questions and I'm also a bit over time, so. But thank you again and let me wrap it up. Enjoy the rest of the conference.