LLMTools and the LLMTool Repository

All right, I got a lot to get through, so I'm going to go ahead and get started. I'm Bob St. Heinrich. I'm a developer here at Wolfram, and today I'm talking about LLM Tools and the LLM tool repository. Hopefully, everybody's sort of familiar with LLMs in general. They've taken over the tech world in the last year. Hopefully, if you weren't already familiar somewhat with the integration of LLMs into Wolfram technology and the Wolfram language, you are after, you know, the first few hours of the conference today. So I'm sort of working on that assumption, and we're gonna, we're gonna build on that and talk about how l. How tools are a big part of that integration with the Wolfram language and really are kind of what makes it powerful and special. What is an LLM tool? You've maybe seen these in Talk already. They appeared in Steven's keynote, but he didn't really go into what exactly was happening. A brief introduction. So tool is just a piece of code that runs outside of an LLM that the LLM can use. In this case, it's Wolfram Language code, a Wolfram language function that the LLM is told about. And whenever the LLM chooses to use that function, it can send a special response to the Wolfram language. Instead of responding to the original question like it normally would, it'll give a special response that'll say, hey, run the function and tell me the output. And then the Wolfram language will run the function and create an output, and then it can use that output as however it wants to create its final response. In a normal conversation, you ask some question, and the LLM just makes up the response using its trillion parameters. But with a tool, you ask it a question and then it calls back, and maybe the tool is something that does math and it's called Calculator Tool. And then the Wolfram language generates a. Generates a response, and then it can use that tool. All right, so, you know, briefly, why do we need tools? LLMs aren't. Are all powerful, aren't they? Well, that's not right. You know, I think I've heard a few times today already people be moaning how bad LMS are at computation, you know, which is kind of strange because, you know, we've grown up accustomed to computers being better at humans, better than humans at computation, and LLMs a lot of times aren't right. So here's just some simple floating point math, and it's off by nine orders of magnitude this time. So like really bad. But Wolfram language is really good at computation. So by hooking up LLM, by hooking up tools to an LLM, you can get sort of the best of both worlds. So the LLM can do what it does really well, the writing. And if you see here, this progress indicator was up and then it went away and then came back. That's that two call process that I, that I was showing the last slide. So. So now you know, if you just ask the lm, it's going to give you something really bad. Sometimes it'll put some nice explanation around this. Explain, you know, really eloquently explaining its terrible answer. If you just ask Wolfram language, you'll get the right answer, but you don't sort of get the nice explanation. And if you ask the LM with a tool, you get the right answer with some nice explanation. And there's some formatting that, you know, we don't really support. But it's superscript, it's supposed to be a superscript. You can probably figure that out and it's all great. This is really showing the point. And don't worry too much about this thing. We'll get to that later. Just see it as a black box for now. How do you work with tools? In Wolfram language, the main function, the main symbol to think about is LLM tool. And this is released into 13.3 and version 14 and it's documented and you can look it up and use it. Here's the simplest use case where you give the tool a name. That's what the LLM calls it when it's trying to use it. You say what parameters the LLM should provide. This one just has a single parameter called param1 and the Wolfram language is going to treat the value of that as a string. And then you give a function, this is just a trivial function called F that creates this LLM tool object. That then is a symbolic representation of a tool that an LLM can use. But you don't need an LM to use it. You can call it directly, which is pretty useful for testing and development when you're making tools yourself. This is here I'm acting like the LLM and providing data that normally the LLM would be providing and just seeing that it works. So now let's move on to like a more interesting case. So here I have a tool called Currency Convert and I give this explanation that tells the LLM how and when to use the tool. And I give an example in here. Examples are always really Useful most of the time. Here you're just going to talk to tell the L, give the element instructions like you're talking to a person. That's good rule of thumb. But giving examples is also really helpful. Here's the parameters. This one takes two parameters, text and currency. Text is interpreted as a string, currency is interpreted as a currency name. Then here's the code of the tool. It's two lines of code. The main thing here is currency convert and string replace. Now I have this LM tool that's more interesting than the dummy one at the beginning. Again, I can apply it manually. So I'm filling the role of the LLM here. We're actually calling the service and I see here I have a value in my string that's in US dollars and I say make it yen and it complies. So the tool works. So now let's see how these things work with an actual LLM. I'm going to make sure that I'm using GPT4. GPT4 is much better at tool handling than other models. It's better than 3.5, it's better than Anthropics from what I've seen. But you know, they're all going to get pretty good at this if I had to guess as time goes on. So now I'm calling LM Synthesize and I'm giving it. I'm using the LM Evaluator option. And within that there's a. There's a property called tools that takes a list of LLM tools. And you'll notice I also have the second argument all. I'm gonna go ahead and start this because it can take a while. So when I give all LLM Synthesize, instead of just giving the final response from the LLM, it tells me a bunch of information about the entire interaction, which is really useful for tools because as I mentioned at the beginning, it's not a simple question answer request response. It's this back and forth. You can see this one's taken a while. I said brief conversation, but brief is a relative term that progress indicator went away. It generated its first response, then it called the tool, the tool ran, generated an output, sent it back to the LLM, in this case GPT4. And now GPT4 is building its final response and it's done. So let's look at what it finally did. Remember, I told it to respond with both the original and converted conversations. So did it do it? Yep. So here's the conversation where everything's in US dollars and Then here's the conversation. You see, it looks like everything is exactly the same except the currency values. So that seems like it worked. So let's dig into what actually happened a little bit with this. Llmsynthesize of all. In this result object, there's a key called tool requests. And I can see that there was one tool request. And you can see that the text and the currency that was in there. And this is just like the position within the full response of where this request happened. I can see what the output of the. Of the function was by looking at tool responses, which again would be like this symbolic object. So I'm pulling out the string output using output, and then we can look at the whole conversation. This is just sort of splitting it into chunks and giving different colors to make it easier to see. So this, let's call this aqua, is sort of the original instructions to the elements. So you'll see it doesn't just start off with my write a conversation about the price of gasoline. There's all of this tool instructions. So it tells that it has tools. It tells us which tools it has. It tells it which parameters it needs to pass and what those parameters should look like. It tells it how to call a tool in general. It says, always use JSON. So there's all of this light green is just about the tools. So that's most of the original request to the LLM. Then it has the text that I put into LM synthesize. Here's the first thing it did. It wrote the conversation. And then it says, I'm going to call the tool. And then, oh, shoot, I'm going to need to exit full screen here because I think this is a. There's a piece here that's missing. All right, here's the tool call. I didn't want to. I think that was off to the right. All right, so here's the tool call and I'll go back to full screen for size as soon as we're done going through this thing. And then here's the response from the tool, the same thing we saw when we were looking at the tool response in the result above. And then here's the final converted conversation. So you can sort of see the entire tool interaction. And it's important to note that when it's. It doesn't always just have to be one tool request, one tool response. It can. The LLM can decide. It can make several. It can. You can give it many tools and it can choose to use just one of them. It can choose to use two of them five times each. You know, it's completely up to the LLM to decide which tools to use and when, if you make them available. And this isn't just an LM synthesize, it also works in LLM function. So here I have a function, an LLM function that can be applied to values, and when I do that, it will make the tools available every time. So this also works in chat Evaluate LM example function. Probably anywhere that LM Evaluator supported. You can include these tools. I'm not going to run this one just because kind of slow and we have plenty to get through. But you can see that I have that. My candy bar jingle, you know, it's Halloween yesterday, so my candy bar jingle has euros for Austria and Australian dollars here. So it seems to work. It's different jingles in these two cases because I'm calling the entire function each time and each time it's writing a new jingle. So it's not. It's not providing the original one. All right, so that's sort of how you use tools programmatically in like LM synthesize and LLM function, but you can also use them through the notebook interface. So I'm going to create a chat notebook, and I don't want this full screen and chat notebook comes with certain tools built into it. And so just, I'm going to say create a visual of a pebble dropped into a pond. I should have told it not to explain too much, but. Oh, well, we'll let it go on while we talk about this. So within chat notebooks, many tools are available by default, and you'll have control over which tools you want to be enabled at any time. And we're going to look at exactly how to control that in a minute when this is done, with its lengthy explanation of everything it's doing. Oh, it didn't even use a tool. This happens frequently. I'm a tell it. Do it. All right, so now it's going to use a tool called Wolfram Language Evaluator to actually run this code and it'll insert it into the chat output here and then it'll probably drone on explaining what it did again. But that's helpful sometimes. So I'll let it. Let it finish talking and. And then we'll see if the animation's good. All right, let's. Let's see. Oh, it did not do well. It did not do well. Let's see. Anybody tell what's wrong with the code? It did this excellently when I was trying graphics circle, plot range. What is plot? This is. I don't know. All right, well. Oh, wait. Oh, it's finally happening. It's finally happening. I see the circle's just growing really slowly, I guess. So it did work. It's done some much better examples with like, bessel functions and things before when I've tried. But you never know how creative the LLM is going to be. All right, so this is using the default Persona and its default tools. I can create a chat delimiter here and create a new chat input cell. This is a tilde, and this is a single quote. Create these cells here. Within this menu, there's add and manage tools. So I'm going to go here. And don't necessarily fall in love with this interface because it could change slightly. But you can see the way that tools are controlled in chat notebooks is by Persona. Every Persona has a set of tools that are enabled when you use that. Persona Chat Assistant is the default Persona. And see, Wolfram Language Evaluator is checked. So that's why it was able to use that in my previous example. Bernardo, everybody's favorite bird uses a whole bunch of tools. And I've enabled Create Notebook for the Seuss Goose. So I'm gonna, I'm gonna, I'm gonna work with the goose here for a minute and show that I can, I can use a different tools, make a notebook with a very brief math lesson. All right. Hopefully it will respect my request for brevity. Ah, the goose is, is very Susan. All right, now it's calling the tool. This tool instead of Wolfram Language Evaluator is Create Notebook. And this is a tool from the tool repository, which we will see in a minute. It's a little bit slow because it's launching a dedicated kernel for security reasons. So that's, that's sort of an implementation detail of some of the tools that the first time you use them in a session, they'll be. They'll be slow. So you see, it worked. It created this. For some reason, the math lesson is not a Seuss Goose seed. So I don't know why not. But you never know these things, and now there's a long way. So the lesson was long, was. Was very brief. But the, the poem explaining what it did was. Is long. Okay, so. Right, so you can use Personas to control what tools are available. If I go back here and I want to add a new tool and I say go to the tool repository here I have the tool repository. Let me see if I Can magnify a little bit. You see there aren't very many tools here yet. This thing is not really released or announced yet. It's live and working and please play around with it. But the content sparse and we need your help with that. We need your help creating great tools that people will love and use. You see Create Notebook here you see Wolfram Language Evaluator. You see Text Retrieval and Text Storage which I'm going to use lately. If I go to one of these now, since I came from that dialog, if I go here, I have this install button, I I can click install and I can go back here and now in this notebook I should have. So here it is. So I could give that one to the goose too and I could keep going and it would have more tools going. But I'm going to get back to my slideshow right so. All right, so we just saw this. Everything works. So I don't need to show you my pre created example, the tool repository we just saw the homepage for. So this is. Yeah, the a new part. If you, if you're familiar with other repositories like the function repository or packlet repository, you should feel right at home working with a new tool repository. So the main function for accessing tools from the tool repositories LLM Resource tool. We saw this right at the beginning of my talk. In the very first example, you just give it a name. But what you get back is not an LLM tool. It's a template object. That's because oftentimes you want to configure these tools. There's two sets of parameters. There's one set of parameters for the human to configure how the tool will work for them. Then there's a second set of parameters for the LLM to fill in to actually run the function. This text storage one takes a parameter location that says where should the text be stored? I'm going to say I want to be stored locally. Then to save time here, I'm just not actually going to use it with an LLM. I'm just going to run it myself. What I get back from my tool is this uri and then I can configure its partner tool, Text Retrieval to also work locally. I can get that back. This pair is really useful because oftentimes there's some big body of text, maybe it was generated during a previous conversation with an LLM that you want to make available and be able to reference with an LLM but not actually put it in the message to the LLM, because that costs a lot of time and money for a large piece of text. And there's a strict limit with most of these elements for how much text can be in there. So instead of giving the text, you can just give it a reference to the text and then you can retrieve it later, and the LLM can retrieve it later as it needs to. So these are kind of, this is kind of a useful pair of tools that you can use together with an LLM. But yeah, so the important concept here is that the tools from the repository are configurable and they have these user parameters and LLM parameters that are sort of distinct. So the LLM parameters live within the final LLM tool and the user parameters live within the template object that surrounds the LLM tool. All right, so, yeah, so if you have a great idea for a tool, we're very eager to, to have you submit it. Let me exit full screen on this and magnify it a bit. So if you're familiar with submitting content to other repositories, function packlet or anything else, hopefully this definition notebook doesn't look too foreign to you. If you saw Eli's talk half an hour ago, there's a definition notebook for prompts to put in the prompt repository. This is a very similar one. You just fill in my tool and then you give a description. Then the important part is here's the function and these hash values represent the values to be provided by the LLM. And then you can add a separate parameter and you go up here and you say template slot named. And this creates the parameters to be set by the human and you can set defaults. That's often good to do, you know, whatever, whatever the default is. I have a definition notebook that I filled out earlier called Simplify Math, which would make just make the Wolfram language function Simplify available to the LLM, which could be pretty useful. I think I might end up actually submitting this one once I've created an icon and added some examples. But you create the function. I have this tostring wrapped around it because I want the full thing to be passed back to yellow. And I don't want to trust. The serializer to, to do it. But there's two sets of examples to create here. There's chat examples and programmatic examples, just like we've seen in this presentation. There's two main interfaces for using these tools and using LLMs and Wolfram language. There's the programmatic interface with things like LLM synthesize and there's the notebook interface with chat notebooks. As soon as I create this, it's available in this chat section. So I don't need to like, I don't need to configure it with, with the Personas while I'm in the Definition Notebook. The Definition notebook is set up to include the tool by default, regardless of the Persona. So I serve as a sort of contrived example where I generate get the LLM to create some mathematical expression and it calls simplify. And all it did for me this time is pull this square root of two out. So it wasn't a great example. But you can see this is just like in a, in just like we saw in the chat notebook where you can see the call to the tool and the output from the tool here. And you should document here the, the user parameters. So the configuration, I added one here that was called time limit. I think I screwed up my casing here. I should make this capital T, capital L. But it's, you know, users can set this up to, you know, go fast, to only go fast or to allow it as much time as it wants. And here's, here's another contrived example where I'm getting the LLM to create a mathematical expression which is then simplified and then you want to fill out metadata. So I actually haven't gotten to this yet. I'll go ahead and do that as we're here. Put in my name. I'll put in some categories like. Oh, let's see. Higher mathematical computation is probably what I want. Some keywords. Math trigonometry simplifies great with trigonometry and I could probably fill that out more. But these, these are for humans, not for the LLM. The LLM won't see these. This is just so other people can find your tool in the tool repository. Related documentation pages. Obviously Simplify. I'd put that in there. Related resource objects would be good to do a quick search of the function repository or the tool repository or the prompt repository for any related resources and I'll put their names in here and then I'll hit check. And this is a very important step. You never want to skip hitting check and this will go through and run a bunch of validation for the fields. Ok, so this is, this is a bug. It's telling me my category is wrong, but it's just a checkbox. That's something for me to fix. That's a bug. I'll report it to myself. This is basically how you fill that when you're satisfied, you'll hit Submit to Repository. It requires a publisher ID like any other repository. If you've submitted to Function Repository, you already have one, you can use the same publisher ID and, and that's, that's how it works. All right, so I've been ignoring any questions that have come in. Let me see. That's basically the end of my presentation, I think, right? Oh yeah. I want to highlight some more information. So additional LLM related talks are listed here. You know, Chat Notebook Basics and Advanced Chat Notebook from. From Rick and Teo are going to be really great. Check those out. If you want to learn about other services besides the OpenAI and Anthropic and Palm, go to the future of API integration. That's really good. Then on Friday afternoon, we'll have a full hour to take all questions and really have more time. All right, let's see. How do I find the questions? They've got to be here some. Oh, I see. Okay. All right. Okay. Big new asked about posting my notebooks. They should already be posted in this session. And Jason asks, how would we include rounding to be more realistic? I guess that's with my first, very first example where I showed floating point Math is terrible. So how would we include rounding? I mean, so. Right. So Wolfram Language Evaluator is going to use text string, I believe, on the output. So you could write your own tool that you know, creates the. The eventually the output of the tool has to be turned into a string. You can do it yourself within the tool, or you can trust the LLM synthesize to do it. In which case I think it's going to use text string by default. So instead of trusting texturing, you could format the output as a string yourself and add as many digits as you want with something like integer string or one of the many functions in Wolfram Language. Anton is asking, can we specify in a prompt that you want a tool to be used? Absolutely. Oh, I should have mentioned that. Yes. For Personas, at least when you create a Persona in the prompt Repository notebook, there's these LLM configuration and there's this LLM tool slot and you list LLM tools here. The UX on this spot in the Definition notebook isn't great. Look for that to be cleaner and to have one of those info bubbles in an update soon. But you list tools here and those will be checked by default. They can be the names of tools that are already published in the tool repository. Just the plain string, just the name. Or they can be custom LM tools In which case they're like only available via this Persona. They won't be available with LM Resource tool. They'll only be available when you're using that Persona. So. Yes, yes, you can definitely do that. Danny's asking, can the LM fill in time constraint two if you pass one in? Yeah, yeah. So it's up to you if you choose to make a parameter, a human parameter or an LLM parameter, but something like time constraint, you probably most is more commonly humans want to configure it because it's their, Their, you know, computation. Anton says using tools is like pushing rope as anything with LLMs. Right, right, right, right. So, you know, the solution should come from the inside. Is that the idea that, you know, we should. Yeah, I mean, that's sort of a large picture viewpoint that I'm not going to. Yeah. We could discuss it for minutes and I have zero minutes. Oh, Eli says I haven't done my handout thing. Right. Well, I'm sure that the conference, the events people will help me make sure that the handout is properly distributed because I. If I screwed up, I don't know how to fix it right now. Handouts. Oh, I didn't hit share. Okay. Ah, Eli was right. Okay. All right. And you know, if people want to leave to go to other talks, because I'm two minutes over, please leave to go to other talks. All right. And then do you have separate curators for Wolfram Tool, WTR and wpr, Right? Yeah. I don't know if WTR is the acronym we're going for Tool Repository. Usually just an acronym emerges organically and it hasn't yet. So the curators overlap for tool repository and Property repository. But you know that it's. It's also an evolving group. People, people are added. People leave. Leave the company, you know, but it's currently. It's an. It's. It's an overlapping group with some distinct members in each and. And then Danny is asking Anton's question and. All right, all right. Looks like that's the questions. So. Right. I encourage you to go to the other talks that I was highlighting here. And I also have two more talks on completely unrelated topics. One about Packlet Repository tomorrow morning and one about the question framework tomorrow afternoon. So if you just love my voice, come check those out and I will. And then Friday I'll be in the workshop. So if you come. If you think of more questions and. Or longer questions, please, please save them for then. Thank. You.