LLM Functions

All right, so over the last few versions, as I'm sure you saw in the keynote and elsewhere, we've been adding a lot of LLM functionality to Wolfram language. So there are going to be several different talks over the course of the next few days about different aspects of LLM functionality. This presentation is hopefully just going to be kind of an overview, talk a little bit about everything. But there'll be several more presentations over the next few days that'll go into more specifics about some of the areas that I'll sort of touch on here. So let's see. So the first thing maybe to mention is, so at least right now all of our LLM functionality is provided through different APIs from different providers of LLMs like OpenAI or Anthropic or Google and and so on, maybe in the future we'll have more support for locally running LLMs, but for now it's all through APIs. So if you want to actually run anything, generally speaking, if you just run any of the LLM based functions the first time you run it, a dialog window will come up and basically ask you to put in an API key from whatever LLM provider you're using. It'll have instructions that sort of explain how to do this. So I won't go too in depth, but in general, if you want to make sure that you have a connection that's working, you can always just run something like this. And if this returns a. Ok, yeah, this returns a service object that says connected. And so that kind of tells you that it's working, that you are connected to your OpenAI account. In this case, I think right now, I think in 13.3, the only provider that we support right now is OpenAI, but I think there are plans in 14 and maybe in a packlet update to 13.3 to support several other providers. So like I think we're planning support for Claude and Palm from Google and so on. So but for now, if you want to follow along, then It'll probably be OpenAI for now. So let's see. So yeah, so the most basic function that you'll be using to contact LLMs is LLM synthesize. LLM synthesize deals with probably the majority of cases where you want to call an LLM. And this is a very simple function. It basically just takes a string and returns a string. So it takes a prompt and then it returns the response from the LLM. So I can write something like LLMsynthesize of, you know, write a, let's say Single stanza poem about a koala. And I'll run that. And it might take it a moment to respond, but okay, well, that's not really a single stanza, but sure. So here it's written our poem here. One thing maybe to note is that there are a bunch of options for this that I'll get to later. But most LLMs are, if you run them by default, they'll be kind of non deterministic. So if I run this again, it'll probably give a different answer. Right, so in this case it actually wrote a single string stanza poem. Right. But again, this function is kind of the most basic thing that will do most of what you want with LLMs. Let's try maybe doing something slightly more specific. So maybe a task that's a little more targeted. So I can write something like, suppose I'm doing a road trip from Chicago to Boston. Name a city that I should stop in along the way. Okay. So if I run this, hopefully it'll give me a recommendation. Okay. Yeah, so it says I should stop in Niagara Falls. Fine. But it wrote me this whole big. This whole big essay here that I don't really. Maybe I just want. If I want to compute with the output, I don't want a whole essay. I just want it to say the name of the city. So I could maybe modify this and I could say something like write only the name of the city and nothing else. I'll run it again. Okay, and this time it said Cleveland instead of Niagara Falls. But fine, so. Right. So now I might be able to use this for further computation. There's actually maybe a better way I could have written this though, which is we have a symbol called llmprompt which essentially calls out to the Wolfram prompt repository, which we've been working on, which is basically just has lots of pre made common prompts in it. So an example of a pre made common prompt is one, I think it's called nothing else. Right. So now we get this template object and if I open it up here, I hope that text is large enough. But basically this shows what the actual prompt is. So in this case it's just a bunch of text that's similar to this sentence I wrote here. It says, you know, only give output in the form of a, and there's a slot and then it just tells it, you know, basically don't write a whole essay, just give me a. A direct answer. In this case, what I want back is a city. So I can just write something like city here. And you see, it will it will fill in that slot with the word city. So now if I want to incorporate this into my call to the LLM here, it's pretty simple. I can basically just give it a list and I can give it, and I can give it this prompt as, you know, an element of that list along with the string that I'm is the main prompt that I'm giving to the LLM. So if I run this, I should get back. Yeah, just, just Cleveland. And if you want to see what this is doing internally, there's a second argument to LLM prompt where I can or sorry LLM synthesize. Or I could say something like prompt text. And this will show me the raw text that's actually being given to the, that's actually being given to the LLM. So in this case, you see there's all this stuff which is all from that nothing else prompt that we, that we added. And then there's a bit of white space and then there's this string here which was the string that was, has the actual question we're trying to give to the LLM. So this is what actually got given to the LLM and then it responded with just Cleveland. Right, Let me see. There was a, there was a question in the chat about the non. Is the non deterministic nature of LLM output a bug or a feature? Well, we're going to, we're going to see later on that it's. You can, you can essentially make an LM deterministic. So there's, there's some options that we'll talk about particularly one called temperature that, that you can set that will make it give deterministic output. But I think in general it's thought that both there, there are lots of applications where you want to be non deterministic. You know, for example, the. Write a poem about a koala you probably wanted to write. You know, you might want some variety in the poems you get. So in that case it might be nice for it to be non deterministic. Maybe you want several poems. The other thing is that I think it's been found that when you set temperature to zero, which is basically the way you make it deterministic, it often actually just gives lower quality responses both that they're quote unquote less creative but often just lower quality like by, by various metrics. So it's I think. Right. I think, I think in general it's good to, to use, use them in a kind of non deterministic way. But. Right, let's see here. Right. So, so in that case, right, so, but by the end here, we sort of have this thing where we're giving it these cities and we're getting back a city. Suppose, though, in this case, ultimately all of our inputs are strings and our outputs are also strings. And there are some programs where that's useful, but there are also lots of cases where you'll have some other program that's not ultimately a program on text, and you want to be able to have an LLM perform some intermediate calculation. So that's where. That's where LLM function comes in. So LLM function is essentially a generic wrapper around LLMs that lets you call an LLM on sort of data structures that aren't just text. So maybe let's do a simple example here. I can write something like, right, I can write a prompt here. Something like, right, reverse this string and then a slot, and that represents this LLM function here, which basically contains this prompt. And now I can apply that function to anything I want. So in this case, I'll apply it to a string that's like, hello there. And this will go to the LLM and then we'll get back this response here. So one way you can think of this is it's just a way of representing a prompt. So just to see what this is doing, this is essentially equivalent to running llmsynthesize of template object and then using this template and applying it to this string here. Right? Oh, Oh, I think I need to actually say one there. Oh, sorry. Stringtemplate. Wrong, wrong function there. Okay, yes, there we go. So these are essentially equivalent. So essentially, LLM function is just a way of having passing templated strings into LLM synthesize, but it can do a little bit more because first of all, you can pass in things other than just strings. So for example, I can pass in something like an entity for the city of Chicago, and this will get automatically converted to a string and then passed in. So here we get back Chicago backward. Another nice thing that you can do is, well, right, so let's maybe try applying this. So we have that example here, right? This road trip, road trip thing we were asking. So we can try turning this into a, a more general LLM function where instead of it being specifically between Chicago and Boston, we can have it between sort of two templated cities. So now we have this, this LLM function here. And now I can apply that to, say, two cities like Chicago and Boston. And now it comes back with Cleveland. So again, now we have a function that's it's not just operating on strings, it's operating on generic computable objects. And we're getting back kind of this output here. We can also have it automatically parse the output. So there's a second argument to LLM function, which is basically the interpreter type, which is basically used to parse the output using parsing stuff that was developed originally for WolframAlpha. So if I write something like city here, you see now it says interpreters city. And when I run this, you see that it'll take Cleveland and it will actually, ideally, unless the LLM is going to write a whole long essay about it, ideally it'll come back with an entity that says Cleveland. Let me see here. Well, given that the LLM is taking a long time, I'm going to guess that it's writing a whole long essay. So I'm going to abort it and we'll maybe try one more time. Yeah, there we go. So, right, so now we have a function that's just, it's just a function on entities. There's, there's no visible feature of this that makes it obvious that it's using an LLM. It just happens to be using an LLM internally. And this is really useful just if you want to do programming that relies on LLMs and build up kind of infrastructure, because it means that you can, you know, you can use LLMs in this, you know, in a way that kind of flows with the rest of your, your code. Right. So far in all the applications we've been looking at, the LLMs have just been essentially taking strings and returning strings. But most recent LLMs don't exactly take strings and return strings. Instead they take chats and return chats. So essentially internally, the way the models actually work, chats are often represented in a way that's extremely similar to strings, but they just provide a bit of extra structure which both makes them easier to prompt and a bit, in some ways easier to train as well. And so this is why, generally speaking, the kind of LLM space has been moving toward chat based models more and more. So all of the stuff we saw before was internally using chat models, but the chat aspect of it was kind of pulled off. But it is possible through or from language to access the raw kind of chat behavior mostly through these functions, chat object and chatevaluate. So essentially chat object is, is used to represent chats. So here we have just sort of an empty chat here and then we can use chat evaluate to basically send a message, post a message to a chat and then get a response. So here I'll maybe write chat evaluate, I'll give it the chat and I'll ask it, you know, hello, how are you? Or something. Right. And now this returns a new chat that basically has my message added to it, followed by a message written by the model here. And as usual, they don't usually have the LLMs are never good at small talk. So you know, it's. Yeah, critically though, this, this, this did not modify this chat here. This chat here is, it essentially creates a new chat that has this extra message in it. So, so this is not a mutable operation. So if I run this again, we should just get a different response here. If you want to build up a chat, you can do that by saying something like this. So here we're gonna, we're gonna reset this chat variable to include this new message and now we can maybe post another message to it. Something like, how can I assist you today? Maybe, you know, what's the weather like where you are? Not that it has anything good, it can really say to that. But. Right, so now you see, we now have a whole kind of back and forth dialogue with the model here. There's actually a convenient syntax that you can use to do this, which is, there's a function, I believe it's called apply to that was added relatively recently, I think like three years ago. And it's written with. And essentially all this does is it applies a function to this here and then sets this to the result. So you can actually write something like this. And this will essentially post a message to a chat and then save it in the chat variable. And this works because this is what's called an operator form. So essentially this here is a function that can take a chat and then return, you know, return a new chat. It essentially just evaluates to the same thing as, as this. So if I, if I start with an empty chat and I write this here, this will post a message to the chat and then I can write another one, you know, what's the weather like? And that will append another message to the chat. Right, so now we have two messages in this, in this chat here. So this is kind of a nice way that you can, you can kind of build up whole chat conversations just whoops, just by, let me see, let's not delete the whole presentation. That's odd. Okay, well I'll just leave that there then. Right, so you can write things like this that let you kind of build up a whole conversation just sort of in one go. So the way that these chats are internally represented in this case, I'm just showing you the result, but you can also extract lots of information from this. So for the most useful is probably there's an accessor called Messages that gives you back a list of associations that just represents all the messages that there are in the, in the chat. So here you can see there's a user message that says hello, how are you? And then there's a response, there's a assistant message that says hello, I'm an AI assistant, so I don't have feelings, et cetera, et cetera. And that's the response from the, from the model. So this is internally what chats actually are. They're essentially just strings, a sequence of strings that are annotated as being, having different roles. So the ones that I think OpenAI supports are where the main ones are user, assistant and system. So user is essentially messages from the user from the person calling the LLM Assistant is usually used to represent messages that the LLM responds with. And then system is basically a place that it's basically for prompting is the main, the main use. So if you want to, for example, change the global behavior of the LLM by telling it something like, you know, giving it information about, like in what ways it's supposed to behave, what should the tone of its response be, and so on, then that's all stuff you put in a system prompt. So actually one way that you can construct chats with a system prompt is with chat object here. So the first argument of chat object, you can give it a string that gives a system prompt. So I might give it something like, you know, I don't know, you are Socrates always respond in your, you know, as an ancient Greek philosopher would. Right. And now its responses, I don't really know what exactly it's going to make of that. But now if I look at its responses, you know, it sort of tries to make it sound more like something Socrates would say. So essentially this system message just sort of changes its general, like it's kind of global behavior. Let's see. So there was another question actually about. Right. Does. Does using ChatGPT through the Wolfram language provide more privacy for the user than using it through the OpenAI website? So. Right. So I believe so ultimately when you call it through Wolfram language, it's just calling the normal OpenAI API. So whatever the privacy policy is with the OpenAI API, that's basically what's happening when you call through Wolfram language. I'm not 100% sure about this. But I think that OpenAI does have a different privacy policy for the API versus on the website. But that's something that you should look at for OpenAI's API. And I think the same will pretty much apply for all the different providers. So yeah, ultimately it's still going through those different providers. And so it's. Whatever their API policy is, I think will basically represent what the privacy situation is. But yeah, so let's, let's see here. Right, yeah, so, okay, so we've got, Right, so, so this, this sort of shows how you can, you know, how you can build up explicit chat. Most of the time I, in my experience, something like LLM Synthesize or LLM Function, which internally uses chats but just in the end turns it all into strings at the end. I usually find that that's adequate. But there are some applications where you do need to take advantage of the actual chat structure. And you know, if you need to do that, then you can use Chat Object and Chat Evaluate. I guess it is also useful for things like providing system prompts, which is something you can't necessarily do easily in LLM Synthesize. Right. So maybe the next thing to mention is so far we've just been, we haven't really talked about what models we're using, what provider we're using, what the settings for the models are, but there are a lot of settings and other sort of hyper parameters that you can set when calling into LLMs. So basically all the functions we've talked about so far have an option that lets you set this and the option is called LLM Evaluator. So for example, if I write LLMsynthesize of, you know, write a single stanza poem about a koala, right? So this is just what we had above. I can now provide another option called LLM Evaluator where I can then give it some other settings. So for example, I could say something like temperature arrow zero. So this is that option we talked about before, essentially temperature. It essentially sets how much randomness to put into it. So if you set temperature to zero, then it'll actually make it deterministic. So if I run this multiple times, it should pretty much always return the same result. And you know, it'll be often very kind of mechanical and you know, if you want creative writing, this is probably not the thing thing to do. But if you want it to return something where there's a single right answer, then a low temperature like this might be what you want. I could also set it to something different like, you know, I don't know, two I think is the highest you can do. This will probably just return garbage, but let's see. Actually I'm going to set another option because it's taking a long time. There's an option max items that lets you set how many tokens to respond with before you just cut it off. So. Right. So in this case, yeah, you can see that it just, with a very high temperature, it just writes basically garbage. So you know, maybe, maybe you can set something in between zero and two that's, that's more sensible for your application. But you can see here that, you know, you can, you can set stuff like this in the LLM evaluator. Another thing that you can set in the LLM evaluator is whoops are things like what model to use. So here this is where you would say something like model arrow GPT4 if you wanted to say use GPT4 to write your koala poem. So let's see here. So it'll take a bit longer because GPT4 is a fair bit slower. But right now here, this came from GPT4. Now, right, there are also some other things that you can also provide in the LLM evaluator. So another one that's, that's quite useful is there's, there's a field you can provide called prompts. And this essentially just gives, well, prompts that will get prepended to anything that you request. So I could write something like, you know, answer as, you know, Socrates. Maybe let's write something that's more Socrates specific. You know, what is a circle? I don't know exactly how Socrates is going to answer that, but. Right, okay, so now you see, it's clearly trying to answer this in a Socratic way of some kind. We can also provide. There are, this is also a place where you can, you can feed in things like LLM prompt. So we can use. There's a, there's a prompt I happen to know exists called ELI5, which stands for explain like I'm five. And I guess it's quite a short prompt. It just says answer questions as if the listener is a five year old child. So we can now provide a couple of prompts here. And now when we ask it what is a circle? It will respond in a Socratic ELI five way. So the reason why things like this are useful though is that you can also set it globally. So for example, I can, let me see here. Well, there's a global variable called sorry L on the valuator, which basically represents the full kind of configuration of the LLM that you use by default. So if I set this to something like this, this basically just represents the default configuration, but with these two prompts, then. Now, any call I make to an LLM asking it, say, what is a circle? Will have these prompts prepended to it. So this lets you kind of set the properties of the LLM and kind of take them around from place to place, which can be quite, quite, quite useful. So. Right, right, yeah. And basically all of the functions, all the functions that use LLMs, they all will look at this LLM evaluator variable. So if you set this, you know, this will apply to chat evaluate, it'll apply to LLM function, it'll apply to everything. And. Right, and then also all those functions also take the LLM evaluator option. So, Right, so, okay, so maybe one, one last thing that you can provide in. Well, one last thing I'll mention that you can provide inside of the LLM evaluator is tools. So basically, LLM tool is basically a way of giving LLM access to sort of generic functions that might be useful for it to make its response. So this is closely related to, for example, plugins, if you've seen plugins in ChatGPT or things like that. So let's maybe just do a very quick example, because I know, I think we're kind of running out of time. So here, let's make a tool where suppose we want to give the LLM access to a function that lets it say, look up the populations of cities. So I could write something like this. So I'll tell it, There's a tool called City Population Finder. It takes a city as a parameter, I think I actually, yeah, it takes a city, a parameter called city, and the parameter is of type city. And then for the body, I'm just going to write city of population. So this syntax is all kind of analogous to how API function works. So now I'll maybe have that tool there. And now if I run LLM synthesize and say, what is the population of Chicago? Or maybe let's do something like this, write a few sentences about Chicago and then I'll provide it with an LLM evaluator that has this tool. And now hopefully, okay, in this case, I'm not sure it used the tool because it doesn't mention the population. I'm going to tell it explicitly use the tool. And now hopefully it will use it. Right, so now you see it says, right, Chicago has a population of approximately 2,746,388 people. And I think that's the response that came from the tool. So basically this is a way that you can get the LLMs to give them access to traditional computation, which is useful in the same way that it's useful for humans to have access to traditional comput computation. Both because there's information maybe that the LLM doesn't know, but which is available through the language, or because there are things, for example, that you might want to compute that LLMs are bad at computing, but which in general are much easier to compute. So I believe also that there's going to be. I think Bob is going to give a talk later today about LLM tool and the LLM tool repository that we've been working on, which is basically what it sounds like. So it'll be a. It's a website where, you know, you can see lots of pre made tools that do lots of common operations. So. Right, let me see here. So let me maybe. I think we're just out of time, but I'll maybe go through the questions here quickly if there are any new ones. So will I post the notebook? Yes, I can, I can, I'll. I'll post everything that I've written so far once we're, once we're done. Right. Wouldn't use a chat notebook if you really wanted a running context. I think they're kind of different use cases. So chat notebooks are for. I mean, in general there's a fair amount of integration between the functionality I've been showing and chat notebooks like both of them. For example, LLM configurations can be moved around between LLM functions and chat notebooks. Things like LLM tools can also be used in both chat notebooks and outside of them and so on. But in general I think they're kind of just one of them is the use case for primarily interactive usage and the other is programmatic usage. So here we're mostly looking at if you want to programmatically call an LLM and there are some cases where you do want to programmatically call and keep track of all of the chat structure. Let's see. Right. Does this use. So I think that the last question is does this use rely on the service connection? So yes, basically all of the LLM connections we have right now rely on the service connection pretty much. And so if you set up the service connection then you should be able to call the LLM. There is also, I think, a few other ways of authenticating so you can look at the documentation for LLM Synthesize, I think has other ways of authenticating like I think you can provide, you can provide an OpenAI API key through system credential, which, which also lets you. Is another way of authenticating. But yeah, yeah, I think also Julio might be giving a talk. I'm not sure if he's going to talk about LLM stuff, but he's giving a talk later also about updates to Service Connect, which I think were motivated by doing a lot of the connections for different LLM providers. So there might be some vaguely relevant stuff in there. Right. Well, I think. Let's see. Okay, one, one last question about the right. Is GPT4 available with the larger context length available? Right. So this is, I think GPT4 by default has a. Oh gosh, it's either 8,000 or 16,000. No, I think it's, I think it's 8,000 token context limit, but there is a 32,000 context limit version of the model. So it's basically again, because all of this goes through your OpenAI account, if you're using the OpenAI model, at least it depends on what you have available in your account. So for example, back here where we specified model GPT4, if you have access to the 32K one, you should be able to just write 32K or, or whatever the actual name of the model is. I don't remember it exactly, but it's something like this. So you should just be able to write that and you'll be able to get access to the 32K model if you have, if you have it through your OpenAI account. So yeah, all right, well, I think that's, that's probably about it given that we're a bit over time and I don't think there are any more questions. So, yeah, thank you everyone for joining and I'll try and post the notebook. I'll try and post the notebook soon.