Chat Notebooks Basics

So my name is Theodore Gray. I'm a co founder of Wolfram Research. I was at the company for, I think it's 23 years continuously. And then I went off for quite a while, like 10 years or so doing other things. But I'm back now and working primarily on chat, you know, LLM, AI related features, particularly related to education. But today I'm just going to give you a basic introduction to chat notebooks and what they are and why they are. So to start, go back a little bit in history. If you think about, you know, the 1980s or so, the way that you interacted with a computer system like Mathematica or like, you know, any operating system shell or whatever is you give it a command, it gives you an answer. So a back and forth prompt response sort of thing. And the way in which that was done is what referred to as a glass teletype interface model, of course, after paper teletypes. And a fundamental property of these things is that the record is immutable. It just scrolls along. It's either actually literally printed on paper or it's going off on the screen. And there's really no way to interact with the past because it's happened. But this is obviously unnecessarily limiting. And you know, sort of the origin story of notebooks was just the idea, well why, why can't you edit the previous inputs? Why can't you just go back and select something in your history and you know, re evaluate it or edit it slightly and reevaluate it. And it turns out if you make one small addition which, which we called cell brackets, basically delimiting which what's input and what's output, then you don't have to select a whole range of input. You can just click anywhere within an input cell and it knows to re evaluate that whole command. And if you've labeled output separately from input, then it could automatically delete the output. So new output replaces old, which is impossible in a plain text document. So that's where notebooks came from. Now it's, and this is obviously a tremendously powerful idea. It's been copied lovingly if inadequately by jupyter notebooks. But you know, it's a very good idea and it works very well. So it's a little interesting to see that, you know, chat interfaces, LLM, you know, chatbot interfaces, are basically glass teletype interfaces in their fundamental implementation. And I think that's because they're sort of modeled after human chat interfaces, namely human, know, text messaging or you know, Other such sort of social media things where the Glass Teletype interface is kind of appropriate because the conversation did happen and it's not mutable, you can't go back. And as much as you might want to change what you said and have that become the new reality. But again, in the case of interacting with a chatbot, you're interacting with a computer system, you can change the past. They are in fact completely stateless. They have no memory whatsoever other than what you supply them each time in a chat interaction. So it's actually a perfect use case for a notebook interface. And so that's what we, we decided to do is we're going to implement, you know, the first approximation, simply an interface to a chatbot, except it's a notebook so you can go back and edit the old inputs, you can rerun conversations, you can see what would happen if you change, you know, prompting or other settings or just edited your input or edit the output because you can change what the, the chatbot said and it will believe you just meant to be a very practical talk. So I'm going to actually demonstrate how to do this. So the first thing is how do you create a, what we call chat enabled notebook? The way I always do it is just say new notebook like that, except I already have one here, so we'll use this one. Now if you do that for various implementation reasons, this is an ordinary notebook and it doesn't have chat features enabled in it. But if you're using a sufficiently recent version of later 13 point something, any sort of current version, you'll have this new icon in the toolbar. If you click that, you get this little enable chat features and you click that and now you have a whole new menu and this is now a chat enabled notebook. You can also do it directly here and we'll get to that one later. You can do it this way, but I don't know, I just always do it by creating a notebook. And what does it mean to be chat enabled? What it means is that, oh, and in the future we hoped for 14.0, but maybe not. So wherever the next version, there isn't going to be a difference anymore. Just any new notebook is going to have this capability. And it was just sort of purely for practical reasons that we didn't do that right away. But in the future this will not be an issue. So when you have a chat enabled notebook, the thing you could do is type a single quote character, which I can't show you me doing that, but I just type single quote. Now we have a new type of input cell which is distinguished by this blue border and there's this icon on the side which is actually a drop down menu with a bunch of choices that we'll get to. And this cell, I'll just say hello. If I type shift return just like a regular input cell, instead of sending it to Mathematica, it sends it to. In this case, I've selected GPT4 as my model. If we look at the menu here, it's going to be GPT4 and we'll get to Personas in a minute. But I'm basically using the default code assistant Persona which likes to write Mathematica code. So let me just do a couple of examples and then we can see, let's say so I mean, again, if I just type, I just start typing let's say sine of X here, right? So this is regular Mathematica input. Now I type a single quote. This is actually variation I have not tried before. So this demonstrates the fact that chat history in the chat notebook world is everything that comes above the input cell, the chat input cell that you're evaluating. And this sometimes is a little bit slow because we're at the mercy. I mean this is going off to OpenAI's GPT4 servers, which are sometimes very busy and sometimes things are slow. But the point is it knew, it knew that I was talking about the sine function because it looked and there we go and it was very sensible and figured out how to make a plot and That I wanted -2 PI to 2 PI might be reasonable plot range. Sometimes it picks 0 to 2 PI. You, you know, it's an LLM, so it's not consistent always. So right now the fact that it saw the sine of X there like this is super convenient because it means I can do this, which like you know what the Star Trek is that, that you can say that something like that to a computer and it actually works. But it also means that anytime you evaluate a chat input cell in a notebook, you kind of need to be slightly aware of the fact that it's going to send everything in the notebook, everything above that cell, unless you've put some blockers in it, we'll talk about it's going to send it to the LLM. And this could be, for example, quite expensive. If you have a very large history, there is automatic truncation of various kinds. It doesn't actually send gigantic output cells in their entirety. But you know, tokens cost money. So if you have a very long history that takes both time and money to process that each time. But very useful if you're asking it to, you know, summarize the, you know, what you've done for the last hour or something. It'll do that. And there are potentially, you know, security and privacy implications. If you have some top secret government code here that you're working on, you know, it's being sent to OpenAI and they will do with it as they please. So that's something to keep in mind. So let's see here. I just want to show the fact that, I mean, let's make this tangent instead, right? And if I now reevaluate this input, it's going to do what I think you expect, which is that it's looking at the new history and it's, you know, it's doing it again with the new history. History in chat notebooks is different from history in a kernel. In a kernel, history is basically temporal. You know, it's like if you go to a different window somewhere and do another calculation and then you go back to your first window and ask for percent what matters is not what's above it in that window, it matters what did you do most recently? But that's not the case with chat notebooks. They don't care about the time in which something was done. They care about the place where it is. So it's the structural position within the notebook. If it's physically above the thing you're evaluating, then people get sucked in. And oh, good. This demonstrates there's two different ways that it can do this. When you ask for code here, it's written the code for me and then it's actually gone and evaluated it. And apparently I don't know what that's all about. That's weird. It's vanishing like that. I actually find this often more useful and sometimes I even tell it, don't use the tool. I don't want you to actually generate the plot for me. I just want the code. Because if you get code and output like this, you can then click one of these things. So this one is most useful. This copies it down below and evaluates it. So now it's just created a perfectly ordinary input cell, which is very useful because now if you want to change it a little bit, like you want to go to, I don't know, 4pi this way it's given you a starting point and not too infrequently, the code that it writes is almost right, but not quite. And it's got some, you know, some little Error in it that's pretty easy to spot, but, you know, it's very useful because it's told you about some function you didn't know about or something like that anyway, so I like to just get the code. I usually tell it, don't actually evaluate it for you. Other people, like Rick, whose talk you hopefully will go to right after mine, he has the opposite. He really likes it to do the evaluation in place and don't look at the code. But we need to move on here. So chat delimiters. So if you type a tilde character, so shift back tick here, that gets you this gray line. And the gray line is. Is a blocker. So if I say that, it's going to say, I don't know what you're talking about because it doesn't. See, this line has blocked it from looking backwards. And as mentioned, these LLMs have absolute amnesia. They are stateless every time you hit shift return, they never knew you existed before. They've never heard of you. They have no idea what you've done other than what you send in that interaction in the moment. So if I do evaluation up here, it's going to know because when I hit shift return here, it retransmits this entire history and here it's blocked. So let's see. See. And now it's going to give me a long story about which, you know, didn't actually work, come to think of it, this exclusions. This may be an example of a mistake because I think that was meant to leave out these lines and it obviously didn't. So there's something wrong with the way it's asking for exclusions. But in general, that's. I think, you know, one of the most useful things you can do with chat notebooks is have them create language for you. So I have an example. Good heaven. Sometimes it's very wordy. We have another Persona which we'll get to in a minute. Oh, there's some questions. Audio is really bad. Interesting. In what way is the audio really bad? Because I don't know why it should be. I don't think I'm doing anything. You have a slight crackle. A crackle? Is it happening right now? Yes. Interesting. I have no idea what it is because there's no crackling sounds here. The fan on my computer is on. No, it's definitely a mic issue, like microphone on the laptop. Well, I don't really have any alternatives, so I guess I'll just. I don't know, I'll try to talk louder or Something so to overcome the crackle. So there's a question in the chat enabled notebook. Is there a way to set the temperature to zero so the LLM always outputs the same thing for a given input? The answer is yes, and you can do it in a couple of ways, which I'm going to get to only crackles when I speak. Well, like I said, I have no clue. I'm just. This is the same setup I've used for a million zoom calls and people have not complained. So I don't know. I don't know what's up with that. Let me. Okay, let me show you an example first and then we'll get to the temperature question because it's related to several other issues. This is an example which I think is. I'm going to make another chat delimiter. So starting straight here, write me some code to sort numbers by their English names. So this is a. Why is it misspelled there? Anyway, this is something where I actually needed to do that for something. And let's see what it does this time because it's, it's fascinating variable how it chooses to do this and sometimes it comes up. Yeah, there we go. So this, this is amazing. I didn't know we had a function called Integer name. It would never occurred to me that we would have such a function or how to search for it. But we do. And if we do this. So 1, 3, 2. Yeah, that's alphabetical by its English name. I thought, like I had thought of all kinds of very complicated ways one could do this that would involve big tables of how to say a number. But the LLM taught me that we just have a function for that. Sometimes when you ask for this code, it actually tells you to go to WolframAlpha to look up the name of each number. And that's obviously a terrible way to do it. And when it does that, like the last time I tried to do this demo, it wrote it using WolframAlpha calls, which didn't actually work. And so I just said, you know, in the next cell here, I said, don't use WolframAlpha. I don't have to tell it here. But that was enough. It said, oh, sorry, I won't use wolframpha. And it wrote this code instead, which it could have done in the first place, but didn't. So that sort of maybe is a lead in to the temperature question. So there's a lot of parameters you can set and you can set them at several different levels. So for example, I can click Here. And so we have Persona, we have model and then we have advanced settings and under advanced settings is temperature. So I could set the temperature to zero here. And this is now affecting only this particular input cell. Let's see what we get. With temperature zero. I think you have to watch out for temperature zero because although it will make it more consistent, it's not actually 100% always consistent people's experiences that the answer is almost always bad. It's almost like you need to let the thing have a little bit of creativity or it won't see things that it should. And so it's a double edged sword. I mean there have been some papers published where they said, oh, we're just going to set temperature to zero because that way our results are consistent but it's really not effective because the results are just bad. So you know, 0.7 seems to be, I don't know exactly why that number, but that seems to be what everyone has accepted as a good compromise between, you know, it's a little bit random but it's also useful and gives you things that you can actually use. Like, let's see, does this actually work? Yeah, no it doesn't because spoken stringform is not a function that actually exists. And you know, this is another thing it'll do. You know, it makes stuff up sometimes. We've actually found, I mean Stephen has been inspired in a number of cases where it's like, oh, that's actually a really good idea for a function. Why don't we have that? You know, it makes up functions that make sense and in this case we do actually have this function. It's just not that, that's not what it's called. It's actually called whatever it was that it got last time. So anyway, yeah, temperature zero, you can do it. You can set that, set it on individual one. If you have a chat block divider, you can set it here for, you know, everything that comes below it here or you can do, go here and you can set it for your entire notebook and you can also, in the preferences, which I'm not going to show you because they're, they're just. This is a development version, it's a little bit flaky. In the preferences panel you can set it globally. So if you want to just say temperature zero global you can. But I wouldn't actually recommend it, I've just always left it at 7. And what exactly that means is fascinating and how it affects the sort of psychology of the LLM is fascinating but we've Only got eight minutes left, so let me then show you next Personas. So let's make another chat delimiter and Chat Input Cell. So Personas are basically prompts that are prepended as system prompts to the data that's sent to the LLM together with your chat history and your new question. So it's basically just a block of code. And some of them, that block of code is quite large, like Code Assistant and Codewriter. They are pretty large, complicated prompts that instruct it in detail how to properly format Mathematica code so that it will be recognized. And so you'll get these automatic things. These happen because the prompting has caused the output to be structured in a certain way. Tells it how to use the tools, sets it up. Some of the prompts are quite simple, like Groot. The prompting is quite simple, and hopefully it will actually work. Yes. So the Groot Persona simply answers iamgroot, no matter what you ask it. Except sometimes it can't stand it, it can't help itself, and it keeps talking anyway because it's an LLM and prompts are only approximate requests. Go back to. So the difference between Code Assistant and codewriter. Code Assistant. It writes you code, and it also talks about it. Code Writer is supposed to just write the code and not explain it. So. This should just give me a block of code and not waste time talking about it. Or not. Oh, good heavens. Yeah, that's not. It's not going in a good direction. Let's support that. Try that again. Slowness, unfortunately, is entirely out of our hands. That garbage spilling out was not. Again, this is like it's spilling the guts of what the response is without properly parsing it. Yeah. Yeah, that's disappointing. I wonder if this is a Persona problem. Let's try that in the Code Assistant Persona, this example, it really works quite reliably. Maybe the code writer has some new bugs in its prompting. Anyway, while that's thinking, let me show you some things about Personas. So we have. If you go to add manage Personas here, you. Oh, probably not until it's finished the evaluation. Yeah. Okay. So. This is where Code Writer would have shut up and let me play with it instead of keep talking. All right, now the menu. Okay, but while we're here. So, I mean, although it took a while, you know, I don't know. I think that it's important to be astonished every once in a while, even when you've been doing this for quite a while, that it's possible to actually just say that and Get a. And, you know, an interactive widget, dynamic object, user interface thing just works. And it's just right there. It just works. This is not something that in my entire life experience I thought you'd be able to do with a computer this simple soon because it didn't really seem like there was anything heading in that direction. And then boom, all of a sudden about a year ago. Yes, okay, all right, fine. Computers can actually, you can just talk to them and they do what you want. And this kind of thing is only going to get more and more powerful. Okay, we're really going. This is supposed to be an hour long talk, but it's really only half an hour. So add managed Personas. This lets you control which different Personas you have. And we have a prompt repository which has. I'm not going to go there because we're going to have time. But this has a bunch of different Personas you can install that do things like, you know, they're good at summarizing or they write in a certain style or they try to explain things intuitively. You know, different sorts of personalities that you might want to have. And then there's tools. So tools are a very powerful thing. They're similar to plugins. You may have heard about the WolframAlpha and Wolfram language plugins that OpenAI has deployed, which help the public web version of GPT not give bad answers for math. Tools are like the private version of that and they can do things like, you know, search the web or search files on your computer. Search our documentation, call WolframAlpha. And the difference between these and the plugin is that these things, if you enable one of them, they're actually going back to your computer. They're running Wolfram language code on your computer as opposed to, on, you know, some public server somewhere, which is both scary and also much more powerful because you can ask them to do things with your files. There's a certain amount of sandboxing, but nevertheless, you know, you want to be careful. You can write your own tools where you get to write the Mathematica code that will be called along with the prompting that will install the LLM when to use this tool. And this is all getting into sort of more advanced stuff. So here's the deal. This talk is almost over. We didn't even get the chat driven notebooks. But at 11:30, Rick is giving, who is one of the main developers of the chat notebook technology. He is giving a more technical talk which is primarily focused on. So, you know, notebooks are not Just a thing that you can use. Notebooks are also a development environment. People have written whole big complicated interfaces. We have all these menus, for example. These are all just notebooks, right? They're notebook objects there. It's programming, you know, sort of front end code programming in mathematical language to drive an interface. Well, you can do the same thing with chat notebooks. There's a bunch of, of new things you can do using chat notebooks to create sort of LLM driven user interfaces as a platform rather than as a, you know, thing that you use just interactively the way that a normal user uses a thing. So as a tool, more so than a, than an application. And if that's what you're interested in, you should switch soon, like in the next two minutes, you should switch over to Rick's Talk. But I'm going to hang around for a little while as long as there are questions and try to answer questions for people who just want to use chat notebooks as opposed to use, you know, as opposed to develop tools using chat notebook programming technology. So let's see, we have some questions. So actually all the questions are about my crackling sounds. Okay, here's a question. Does the Persona setting wrap the input in some kind of special prompt guiding the LLM how to respond? Yes, that's what a Persona is. That's in fact the only thing that Persona is, is a prompt and it's prepended and it's sent. If you're familiar with the way that these things work. When you send text to an LLM, you can label it as what the role of the text is. And the role can be like system. And that is then that means this is to be taken as a commandment. You are to do this. The system is ordering you to do this. You can send it as user text, which is this is what the user has typed. You should look at it and respond to it. Or you can send it as assistant, labeled as assistant text. And that means this is what you said. This is what the LLM said. And in order to have the history for the LLM to consider, you need to send it all of that stuff, including what it said. This can be used in various ways, both useful and nefarious, because you can just change what the LLM actually said. What you send back as history doesn't have to be what it sent you. It can be whatever you'd like it to have said, and that will cause it, that will actually influence its future output. But the Persona is simply a system prompt that's sent first and you can see the system prompts in the prompt repository. If you look at the different Personas, you can see them there, you can also see them. So an interesting thing, if you say percent after a chat evaluation I think this will work. Yeah. So the value of percent after you've done a chat evaluation is what's called a chat object and a chat object is a sort of mathematical language interface. So I should mention all this kind of chat stuff. You can do it using kernel functions as a language thing as opposed to as an interface thing. And chat objects are used in that whole set system. I'm sure there's other talks about that and they let you see, you know, in complete sort of code detail what actually happened. By default their display form doesn't include the system prompt. But I believe if I say I'm going to say hey percent I'm going to say object equals percent evaluate. So if I say object I think yes. Ok, so I just asked for the messages property of this object. This will give me everything and that includes here role as system content is boom. Here's the code assistant prompt and you see it's quite lengthy and a lot of it has to do. So these are tool usage prompts, these are telling it how to call different tools. Actually most of this the prompt is the tool usage and then there's examples saying if the user asks this kind of thing and LLMs are remarkably good at generalizing from specific examples sort of by induction. Nobody knows how that works. I don't think so. Anyway, anyway, so this big long prompt and yes, you can see it all, you could modify it, make your own and yeah, this is what you can see through the whole says history. Let's see, can you tell fascinating aspects of temperature? Well, yeah, the temperature is a fascinating property. It goes in the haze of OpenAI models, it goes between 0 and 2. Who knows why. It's almost as arbitrary as the Fahrenheit scale. But it, you know, it's technical definition is these things are looking at the probability of the next token and if you set temperature to zero, it will use whichever token has the highest probability of coming next. According to its calculations, as it's generating output linearly one token at a time, it looks like what is the most likely possible token. I have to pick that one. With a higher temperature it will randomly choose between all the tokens that have at least a certain threshold likeliness and why it is that that causes it to be more creative and give better answers often like you know, not, not Just more creative and interesting or random answers, but actually better answers. I don't know. It's. Yeah, that's. The volumes have been written about this and the people who, if anybody actually does know, they're not talking because it's, you know, it's like a very deep part of the system or sort of. I don't know. Stephen has some blog posts about it. It's fascinating. It's messy. It's the new world. Yes. So somebody mentioned Steven's blog. His blog is quite good in explaining some aspects of what temperature means. The thing that I think is most surprising about these lms is that you talk about, well, most likely token. And it sounds very lexical. It sounds like, you know, you're just generating strings of text by likelihood, which obviously can't give you anything. You know, that involves the kind of creativity and insight that you see coming from these things. And I think that's because this tokenization and probability, whatever, it's applying in sort of concept space too, not just lexical space. And deep inside, in these many layers, you know, it's. It's moving through stages of conceptualization and abstraction rather than stages of lexical text generation. So, yeah, it's. It's complicated. I don't know, difficult, but amazing and worth looking into. And Steven's blog is quite good question. Can you make your own Persona? Yes, you can make your own Persona there. If you go to the prompt repository, there's a template notebook. This might be a good thing to. I think Rick might talk about that in his talk. I'm not sure the details of how to do it. There's a sort of a template notebook that you get and then you fill in what you want the prompt to be. And you know that this is. This gets into prompt engineering. The question of, you know, how do you. You know, how do you get. How do you write a good prompt? So, you know, how do you write a good prompt? Well, there's different. There's different philosophies on that, I think. I think Stephen's assessment is one that I largely agree with, which is that it's like persuasive writing. You know, if you're trying to convince somebody to do something, that sort of language seems to be relatively effective as language for prompting. You don't give it partial sentences or just kind of staccato commands or whatever. You give it clearly written exposition of what is the correct thing to do and what is not the right thing to do. Give it examples to follow. So basically the same kind of thing that would work with a graduate student. Maybe it works with the LLM too. You have to be clear, don't muddle things, make sure the ideas are expressed well and somehow it picks that up and those turn into nice clean vectors somewhere in concept space that it can follow. But it's definitely a bit of a black art. And sometimes repetition helps, sometimes it doesn't. Some people seem to be better at it than others. I don't know. I just keep typing different things until it starts doing what I want and usually get frustrated along the way. Let's see. Yeah. Oh, so I can just mention chat driven notebooks is something that I had not go back to. Sharing the screen, not mentioned. So let's go back to. So the difference between a chat driven chat enabled notebook is what we've been doing. It's like you're basically, if you start typing, you get a Mathematica input cell. If you type single quote, you get a chat input cell. Chat driven notebook. It's just by default if you start typing. So I didn't type anything extra, I just typed a letter and. It creates a chat input cell. I think it's also the default. Yeah, the default Persona is plain chat as opposed to codewriter. And plain chat basically just means minimal additional prompting beyond what the raw. What you would get on the web if you just went to the web interface. Little tiny bit. Just so that it knows about the fact that it's in a notebook environment and there's certain things that it can do. But yeah, so now it's gonna talk about dogs. I should have said, no, you're talking about dogs, and tried to confuse it. I'm. I'm not sure what the point of chat driven notebooks is. I think there's a possibility we're going to get rid of them because it doesn't seem that they're perhaps as useful as we might have thought. What I thought might be a very useful variation is the idea that at some point and for some users you might stop really or never start thinking about this place as a place where you write Mathematica code to solve your problems. Rather, it's a place where you interact with a chatbot who solves problems for you and you rarely if ever actually interact with the code itself. You just interact with the chatbot which is continually writing and modifying and improving code for you. In that situation, a chat driven notebook where the default input is chat input would make sense, but the default Persona would be code, you know, code writer, not, not plain chat. So we Might, you know, and I think the. Right now it's the system is like the world's not quite ready for that. It can write code, but it can't write code well enough. There's enough situations where the code just doesn't quite work or doesn't work at all. And you kind of end up having to drop down eventually into the level of the code itself and make some modifications. Not that that isn't useful because it often gets you very close. And in my case, where I've been using Mathematica for 35 years, it shows me functions I didn't know exist and techniques I didn't realize and new functions like I didn't know we had an up to function which is tremendously useful. Or what are the arguments to partition? There's so many of them and they're so confusing and it'll get that stuff for you and, and you don't have to try to go to the documentation. It's like an automatic example generator that gives you an example that's very, very close to the thing you actually wanted. But then you do still have to go and do a little bit of extra work. And so the world is I think, not quite ready there for a purely chat driven interface where you don't actually use code at all. But I feel like it'll get there and at that point maybe a chat driven text, you know, chat only sort of notebook will make more sense. Any plans for adopting Debating. Oh, between different LLMs in the chat notebooks. Yes. So that's something again, I didn't get to. That's something I thought would be really cool. It's almost there. So I mean you can have different Personas, you can switch and get a different answer from a different Persona. What I would like to be able to do, we had this idea which is that you can say T. And then the name of a Persona. So let's see, what are some names that I could use? 19th, that's too, I'm going to say group 19th century novelist is too hard to type. So this theoretically now sends the conversation to the group Persona. But it may not have worked because he's not supposed to say that anyway. What I really like to have work, but it's not implemented yet is that you'd be able to say more than one different thing. Right. Like you could say one thing, you know, another one, another one and then get back simultaneously or maybe sequentially three different responses based on three different Personas. And that seems to me like that would be very cool. And I hope that we will go ahead and implement that at some point. You can just give a list of the different Personas you want a single input to be sent to. Another thing that I think would be very cool, for example, and not hard to do would be multiple temperature answers like give me this answer, but at several different temperature settings. Ultimately though, if you're trying to do certain things like that, you really should be using the language version of these things like LLM synthesize, you know. Well, I'm just going to say hello because it gives you a short answer so you don't have to use chat notebooks for all of this. You can do all of these kinds of interactions using programmatic constructs, you know, symbolic programming constructs. And here, for example, there's, I don't know, I don't remember the option names, but you can give it, you know, options for what model you want to use. Let's say you want to try this out on GPT 4 and 3.5 and 2, and you want to try it on Bard and, you know, whatever, you know, five different models. Well, you can just map, you know, just say map LLM synthesize onto, you know, some options here, onto a list of, a list of models or service connection specifications or whatever you have. You can use all that stuff. A very simple function that I feel like probably somebody has written is how about take an input, there's some question, send it to the LLM with high temperature, like 1.5 or something really extreme. Like it's. The thing is on acid it's making stuff up 10 times, come up with 10 wild answers and then send each of those answers, or send all 10 of those answers to the LLM with a much lower, more sober temperature and ask it to evaluate which of these is the best answer. That's a very simple function you could write symbolically using things like llmsynthesize and some of the other low level functions. And maybe that would be really good because that's the LLM equivalent of a brainstorming session where you get people together and say, well, just throw out ideas, just come up with whatever you want, no filter. And maybe most of the ideas are stupid, but something brilliant comes up because of the randomness and the sort of freedom of not trying to be too sensible. But then you need to look at all those ideas, write them down and throw out the crazy ones and you know, and in the cold light of reason, pick something that's better. Well, you can do that, but it's not something you probably want to try to do, you know, in a chat notebook as such, you do it in language, that kind of thing. Possible to use different modal ML systems like having GPT4 write a beautiful description of a scene and then send it over to mid journey to synthesize it. Yeah, yes, absolutely you can do that. Probably the easiest way to do that kind of thing if you want to create a pipeline is to do it programmatically. Use LM synthesize and then there's image synthesize, I think it's what it's called. And all these have options. You can choose which model you want to send it to. A thing that I'm working on is an automated tutoring system. One thing we do there is we have, you know, after the student has done some input, we ask, have a sort of a side conversation with the LLM using the, the fanciest one we can get, GPT4, which is much better than all the others ask it, you know, look at this history that the student has had with the chat. Does it seem like they understand the topic and GPT4 is sophisticated enough to be able to make a reasonable judgment about that? The prompting asks it to, you know, write a couple sentences about the student's level of understanding. But then the software needs to make a yes or no decision, like should we now, you know, suggest that the student move on to the next topic because they've understood this one and that's a much easier task. It just needs to look at the sophisticated summary that's been generated by the expensive high level LLM and just kind of say yes or no. Did the student understand it based on this summary? So that, that I'm sending to 3.5 Turbo, which is 10 times cheaper and also faster. So you know, in a programmatic environment you have complete control over this. You can mix and match things. I mean, this is a very powerful concept, the synthesis of symbolic computing. With LLM style computing you get, you know, a thing which is much more powerful than either of them alone, precisely because you can use the sort of the precision of symbolic programming or you know, traditional computer programming, type programming. And then, you know, call out to an LLM like you have a certain structure, you have an interface, it's implemented in, you know, thousands of lines of traditional computer code. But at some point you need to decide did the student answer this question correctly? And that's completely outside the scope of what you can plausibly do with pre LLM technology. If the question is so. One example I have is ask the student, describe exponential notation in your own words. Just tell me what it is. There's never been a system in the history of computing that can judge that answer effectively until GPT4. I mean, not even 3.5 can do it very well, but 4 can do it pretty well. And that's amazing and incredibly powerful. But it's one little slice of functionality that's fit into a traditional piece of software engineering that's implementing a courseware or teaching system. So it's this combination that you're asking about that is, I think, very powerful. And now I think we're well, well over time, and people should probably go on to other people's talk. Go back to the camera. I'm going to wave goodbye and thank you all for attending.