Advanced Chat Notebook Use

Okay. Hello everyone. I'm Rick Hennigan, developer at Wolfram Research. I am the main developer behind Chat notebooks, which hopefully you just heard about in the previous talk. But I will be going over the more advanced topics of chat notebooks, primarily focusing on the developer perspective, people who want to build new applications using chat notebooks as kind of a backend. So the topics I want to go over are all really focused on programmatic aspects of chat notebooks, like how, how can you generate them programmatically, how can you write code that interacts with them, how can you extend the behavior of chat notebooks by adding new features to it or replacing existing chat notebook behavior? Maybe it does something that you don't like or you wanted to do something slightly differently, things of that nature. Then time permitting, I'll be talking about LLM tools with chat notebooks, some of the more advanced topics of how those are integrated. So in the handout section there's a notebook. You can download this notebook. But one thing you'll want to do if you want to follow along with these examples is install this pre release version of the Chatbook packlet. As a bit of warning though, it is untested so there are likely to be bugs. But if something isn't working for you, you can always just pack that, uninsolve that and go back to the usual one. Just so you know, if you wanted to try evaluating these things, you'll need that first. Okay, so the first bit here is creating a chat notebook so with no arguments is basically equivalent to just create notebook with chat as the first argument. Not very interesting, but there it is. You can also specify the initial content that you want as a list of cells, like so. And it can also convert an existing notebook object into a chat notebook. So here's just a notebook in example data and it's also a chat notebook now. So another reason why you might want to use this instead of just create notebook is that it has a whole bunch of options. I definitely don't have time to go over all them, but I'll highlight a couple of them here. Basically, a lot of these things you can set through the user interface already, like what model you're using, things like that. But there are plenty of things that we don't have a UI component for yet, so you can programmatically access those things. Here's an example of something that you can control through the ui, but if you want to have it pre configured for a particular notebook, you can specify that here. This one I said assistance goes to True. That basically just corresponds to this checkbox here. Do automatic result analysis. If you're not sure what that is. That is this type of behavior where it'll immediately jump in if it thinks you did something wrong and try to help you fix it. I'm going to cancel that. Don't want to wait for it. You can also specify things like the model element evaluator. This using anthropic I believe right now requires 14.0. If you're on 13.3, this example probably won't work for you. But this is an example of something we don't have away in the UI to select these yet. That's coming. But for now, if you want to do this, you would need to do it through Create Chat Notebook with this model specification. Yep, that's Claude. Okay, so there's a whole bunch of other ones here. I'm not going to go into too much detail here. I just want to provide this in the downloadable handout notebook. So if you want to play with some of these things and I put a little bit of explanation as to, you know, what each of these things are if you wanted to try it. But you can, you know, for example, provide tools that are just specific to that notebook that other chat notebooks wouldn't see. You can specify a Persona that is unique to that notebook. You can also just give any other arbitrary notebook options if you want to. So this is using that unique tool that I specified for and know that pirate Persona. Let me let this run a bit. All right, that's enough there. Okay. Yeah. So it created the image and one of a couple of things to look for. So you see I have four cells here, but I specified chat history length was three, so it only took three cells. Also I told it not to merge messages. Normally these three user messages in a row would be combined into one, but that was disabled here. So just some examples of kind of customizing behavior through options there. Okay, so there are some functions in the chatbook context that are exposed and available for you to use. There's a lot more than this, but there's just some that I want to highlight because I'm using them in examples here in the interest of time, I can't go through all of them. The chat cell evaluate basically takes a cell object and you can evaluate that cell as a chat input. Current chat settings is similar to current value, which you might be familiar with if you've done some front end programming. But it has some extra special behavior for things like chat delimiters. It can figure out what settings you've set locally to that chat section. Okay, so if I have a chat notebook here, I can grab a cell object evalu. So just programmatically, basically equivalent of the shift enter on that, You would also just be able to use notebook evaluate for that particular example. And it would just work because chat input cells are evaluatable with SHIFT enter. But one of the things I want to point out is why this is different. I'm actually going to reduce the size of this notebook so I can switch between them here. There's no chat input cells in this one here, but I can still send a cell as chat input using this function. I just sent this output cell. I sent that as my chat input and it figured out you didn't do this correctly and it talked about it. Okay, so current chat settings is the other one I wanted to talk about, which basically gives you all the relevant settings for a particular scope right now, with no arguments, basically just saying give me all the chat settings in the current scope, which by default that's just going to be the front end. So these will match up with the options of that curry chat notebook that we're looking at before. But I also can ask for a specific value and it'll actually resolve that as well. You could see, for example, Where's model? Oh yeah, model here. That's something that has a delayed evaluation. But if you do current job settings for a particular value, it'll resolve that. Here's a notebook where I'm specifying a model. It's different from the default. See, my default is GPT4. So now this one is GPT 3.5 Turbo. I can specify that notebook object as the scope that I wanted to check. You can also do this for cell objects, front end session or dollar front end. So one of the things I want to point out here are chat delimiters. So in this one, so I just set for this delimiter that the model is GPT 3.5 Turbo 16K. So now both of these are using that setting as well. But I'm going to change this one to use GPT4. Okay, so now when I look at that list, I see that the first one inherited by the notebook setting, the second one I specified manually to GPT4, and then the last one was inherited by this chat block thing. So it's something that current value would not give you because that would just go straight in the order of cell object. Then look at notebook object, then front end settings, things like that. This is what you would use to Figure out for a particular cell, what are the settings that you should be using? Like I said before, default scope is front end. And you can also modify settings just like current value. You can just use set with equals here to change the value. Oh, did I close it? I did. I closed the notebook already. All right, ignore that setting. But you can also, if I put dollar front end here or just omitted it, they're basically the same thing. So I'm setting right now globally. I've now changed my default model. So even, even if I were to close Mathematica and start again, it would remember this setting. And you can change any of these settings that way. You can also restore them to their defaults by unsetting itself. Okay, I'll get into the questions more towards the end, but okay, so the other thing I want to talk about is adding behavior to chat notebooks. So this is situations where you just want to do something extra that chat notebooks do. You don't want to change something that's already doing. You just want to add some extra evaluations at some point during the chat evaluation process. So far we have a chat pre and a chat post handler which should hopefully be self explanatory when they run, but that basically lets you inject some extra code to run at those points. And it's in the form of a function that is passed some arguments or pass one argument. That's a big association of data that's relevant to that particular chat evaluation. I would recommend this as just a way to see what's available as arguments when you first run these things. I'm creating a chat notebook with handler functions using both of these I just talked about. And I'm just going to. I'm just going to echo them. So I'll say hello, I can see chat pre gets this association these values. So you get your settings that it resolved at the time and event name. You get a list of the messages here, the cell object. So just some stuff that might be useful for whatever code you're running. Chat post gets a lot more information because it's much further along in the evaluation. There's more stuff that has come about there. Okay, so here's an example of how you can hook into these to do some real interesting things. So I'm not going to go over all the code here. It's actually not that complicated. But the main part to look at here is just the prompt. I'm basically calling LLM function with this prompt and I'm just using that association that's passed in Chat post to generate a title and then modify the notebook's title using that. So handle functions chat post, update window title. And I have it set so after three messages it will do that automatically. So here we go. That's pretty cool. Okay, so it should be running my post evaluation now. Oh. Although you probably can't see that because it's very small, but it did change the window title. Well, from tech conference chat, the code looked at the contents of what I've done so far and basically summarized it into a window title. That makes sense. Another variation of this is you could automatically give a file name and save it based on give it a semantic file name based on what you're doing. Which that's actually something that I've been doing a lot lately and it's incredibly helpful. Let's see the other example I wanted to show. Suggestions bar. So again, I'm using chat post for this one. Similar idea. This one is more complicated, but it's mostly just styling stuff. So. Oh, this did this looks better at 100% magnification. But let's see. Tell me some cool coding tricks to show off at the conference. How do I create stunning visualizations with the function? This one I put together quickly as a demo for how to use this and I quickly realized that it's something that I probably want to polish and just put into chat notebooks by default because I've already found it really useful. This is the one I just threw together is kind of a quick hack though and needs to be cleaned up. So the way that worked is I basically said look at the chat transcript and come up with some follow up suggested prompts. I just specify it should be short, it should be relevant to the topic, things like that. And this could probably be improved a bit as well. But for a quick demo it actually worked even better than I was expecting. Those are examples how you can use chat pre and chat post to inject new behavior into chat notebooks. The other thing I wanted to talk about is modifying the existing chat notebook behavior. There are a lot of decisions that were made and a lot of heuristics for figuring out what to do with cells, for example, like output cells for example, that contain giant images. What do you do to convert those to text to send to the LLM? Because the LLM is a pure text based thing yet there are multimodal models out there and GPT4 does have a version that accepts images, but not yet through the API. So we don't have access to that through chat notebooks yet. But in general, when you're using chat notebooks, it's just pure text. So things like images just have to get shortened somehow. And it's true for a lot of other notebook content as well. It can get pretty verbose when you open up a notebook file in a text editor, for example, there are a lot of heuristics to figure out how to convert these to strings. And maybe the heuristics that we chose aren't right for your particular application. Maybe it's trimming something out that you think is important. You might want to change how it converts certain cells to text, for example. Or maybe you want to change how it's doing the formatting of the output right now. By default, it'll format basically by using markdown conversions, things like that. The underlying functions for these things are exposed in the chatbot context. Here it figures out, like chat input, that's a user message, chat output assistant. And it also does like this automatic formatting stuff. So that's the markdown syntax versus a section. And things like input cells get triple backticks to indicate that they're code, things like that. If you want to change what these do, you can right now, instead of handler functions, because these are kind of a different category where they're changing things, this name might change. This is tentative, but for now just calling it processing functions. It works the same way. You give an association of functions by name. This one is basically just converting the cell to a string. Let's look at the chat that was sent. Yeah, so we can see that it just sent the actual cell expression, which isn't as useful. It's just a demo to show how you can change. So that is not using the default built in behavior of that sort of markdown conversion. Another example is maybe you don't want it to grab stuff in your history that is not a chat cell. So that's something that some people have asked about. How can I make it ignore certain cell types by default? For example, like this is one simple way where you can give it a definition where if it's a chat input or chat output style, do the default thing that you would usually do otherwise, do nothing. So nothing will disappear when it ends up in a list. How that works out is I evaluate that because I said it returns nothing for other ones. This section cell basically gets dropped from the chat history. I just got the hello and the response. That's how you would control what types of cells to include. Here's a bigger real example. This will probably become obsolete once we have the multimodal GPT4 available through the API. But it's interesting at least to show. For now, I'm basically just My custom cell to chat message function is going to look at cells and if they have graphics box or graphics 3D box, it's basically going to send to a hugging face model for this one here to generate an image caption. And. And I'm going to use that to create a piece of text that explains what the picture is, rather than just showing some shortened graphics box expression that isn't meaningful. But I'm also including things like metadata and any potential errors. So this is not a multimodal model. This is pure text. But it knows that I'm looking at a golden retriever and a bird because I use another model to generate image captions for them. I think I had this one already evaluated to save time. Yeah, I'm not going to go through all these to save time. I have one that I've already done. But yeah, another thing that makes this useful is, you know, I gave it the ability to figure out. So when you mouse over this thing, I can see it says an improperly formatted option was encountered, etc. Etc. It tells you information about it by default. It wouldn't see that just by looking at the graphics box. So it'd see something like this and not realize that it's incorrect. It doesn't see pink boxes. But what I just did with this function is I gave it that information. I thought I. I guess it didn't mention the error, but it did figure out that it was an error and how to fix it. Okay, Okay. So can we say there's a chatbook feature enhancement framework in place, A very, very early version of it? Yes, but that is the direction we're going in. So the other thing that I wanted to show, one of the things you can override with your own function is the formatter for output. So you've probably noticed that stuff gets automatically formatted in certain ways. This is the function that does that. The fact that this looks slightly transparent is just because this is not a chat notebook style sheet. But one of the things that some people have run into is the LLM sometimes has trouble following all its formatting instructions. Particularly one that I've been trying to prompt it to be better at is properly escaping dollar signs in text when it's not talking about math. So this, this denotes math formatting, right? That's how it works in Markdown. But if it's saying, you know, like $22 here and $35 there and two bits of text. Everything in between those dollar signs gets interpreted as math. So if you're not dealing with math in your output, and you know that maybe you want to override the behavior of this thing to not do math formatting, you can pre process these so that it won't do the math formatting. So actually, in the interest of time, not gonna show this one, but this one, it just reverses the string for the output instead of doing formatting. So it's not a real example, but you could try that if you download the hand handout. This one, there's not a whole lot of time, but I just wanted to show. So the prompting, maybe you got a glimpse of it is. I just asked the LLM to act like a Wolfram kernel, but I added some other stuff too. So this is, this is a kernel that works in every conceivable way to the default one. But it's a good example of how to use some of these things, some of these options that I've talked about, it's kind of. It puts stuff together to make this work. And you would quickly realize that this falls apart very badly because an LLM can't actually compute things any better than a typical human could. So if you start asking it for complicated things, it falls apart. But it's a fun experiment to play with that. Running short on time. But I did want to mention some stuff about tools. So the default tools you've seen probably seen already. So like there's a evaluator, there's a WolframAlpha tool documentation and a documentation search tool, etc. Etc. So these are just the ones that are built in. But one of the things that the default tools have is if I toggle formatting here and just see the raw text, you can see that there's no actual image in this text. So it gets one of these URIs here. And let's see, just kind of skipping ahead to go quicker here. Right? So you get these sort of URIs in here. I've exposed a function here to let you convert those back to the original expressions. And I would recommend, particularly if you're writing an LLM tool for LLM tool repository, this format tool response in particular, I just wanted to point out, and I'm kind of skipping over this, but I just want to point out, to look at this a bit and perhaps utilize that if your tool outputs anything that just will not format well as text, like plots, for example, if the output of your tool has a plot you want to send that to the LLM. I would recommend using this and then it'll automatically just give a sensible output to chat notebooks that gets formatted nicely. Yeah, I don't have much time to go over all the details there. Last thing I wanted to point out is that the chat notebook code is open Source. It's on GitHub, you can look at it, you can submit issues there. If you find bugs and you want to report them there, that is a great place to do it. I will read them and hopefully fix them very quickly and you can track them and see if I'm working on that issue, for example. But also it doesn't have to be just bugs, some of these things. Like I said, this is just the very beginning of extending this as a framework for developing new things. One of the things we really need is feedback about what types of things you would like to see or if you have ideas for something, things like that you can create an issue on GitHub for that where it's a suggestion or feature request, something like that. And that's a great place for it. Let's take a quick look at the questions here. It looks like Bob has been answering questions is good. Which do you think produces better wolfram language code, GPT4 or Claude 2? So I've been using GPT4 a lot longer, so it's hard for me to say, but from the limited amount of testing I've done with Claude 2, I still think GPT 4 is a little bit better. But Claude 2 I think is definitely better than GPT 3.5. Like, I think it's. It's right at the threshold of being really useful. Like GPT 3.5 I consider kind of in the novelty category. It. It's kind of superficial and it looks really neat at a glance, but for doing real things, I think kind of GPT4 is like kind of the threshold of where we really start caring about what it can do. Oh, so someone else is asking if you have an OpenAI API key to use with Mathematica, do you also need to purchase the 20amonth chat GPT plus that? No, you do not. So the OpenAI API and ChatGPT plus, those are two separate products from OpenAI. So ChatGPT plus is what lets you use their web interface. You would have access to things like the Wolfram plugin and whatnot. But if you're just using it from Mathematica or Wolfram Desktop, you don't need chatgpt plus. You just need an API key. So you create an account, you do have to give some payment information for that, but it does not have that subscription. You don't need to pay a monthly fee for it. It's just kind of pay as you go type of thing. Can reinforcement learning be used with LLMs, meaning the LLM learns by talking to itself and the Wolfram plugin? That's certainly possible. We're not, we're not doing that in the context of chat notebooks, at least not yet. We could probably create like a fine tuned model and you could certainly do something yourself and create a fine tuned model. But yeah, that's certainly possible. See, How can I make extensions with my own LLM services? So there is a talk, I believe, I think it's tomorrow, Julio, about the future of API integration. I think called something like that. That would be the place for that type of question to get answers. Are there any plans to enhance Wolfram language? Is Jupyter kernel to have features of the Wolfram language chatbook? That is not something I can answer because I have not worked on any of the Jupyter stuff. But I believe the Jupyter support is on the Wolfram Research GitHub, so you could probably ask there. So if there are no other questions, I think think that about does it. Okay, thank you everyone for coming. And one last thing to point out. There's a LLM workshop tomorrow where you can ask more questions and I'll be there as well. So if you do, you know, want to discuss some of this stuff more, that's the place to do it. Okay, thank you.