Chat Notebooks bring the power of Notebooks to LLMs

Can I get a clear indication as to whether I'm currently live or not? Okay. Apparently you can hear me and see me and hopefully you're seeing my screen. So. My name is Theodore Gray. I'm a co founder of Wolfram Research many, many, many years ago in 1987 or 88. And I'm the original developer of the notebook interface for Mathematica. I have been away from the company for a while doing other things, but I'm back now and very excited to be working on the project of seeing what happens when you integrate our notebook concept with LLMs, GPTs and such like. I wanted to first give you a little bit of history of how these sort of, how these interfaces developed as kind of motivation for how we think about this. Way back when, the standard way in which one interacted with a computer system that you know, would take input and give output with what was called a glass teletype interface. In other words, it's a user interface which carefully replicates all of the limitations of a paper teletype machine. So I'm showing here like this is what you know, what Mathematica looked like back in 1988. You have an interface, you, you know, you, you type, it types back at you and it's completely not editable because this is pretending to be paper. And this has obvious limitations like you know, if I want to do the same calculation again, I have to type it all over again. Or maybe you can copy paste if you're lucky. So there are some clear improvements. There was a program called Xcode which is Apple's development environment, which at the time had basically a textual interface to the shell. But you could edit it and you could select text within it and you know, re evaluate a given line. You could have like multiple lines of input. You could select one and evaluate it and it would place the output below in that document. But then you could select it again and evaluate it again. You could edit it. But there were problems. So for example, because it was a completely blank plain text document, the system had no notion of what's input and what's output. So if you had a multi line input, you would have to manually select the appropriate number of lines before evaluating it. And it didn't have any way of deleting old output because it didn't know what was output and what was not. So you know, my first idea was, well, clearly we need to mark the difference between input and output as different zones. Like we originally had like colored bars. All except they weren't colored, they were textured black and white patterns along the right hand side to demarcate. This is a zone of input, this is the zone of output. And that immediately lets you automatically evaluate all the lines that belong to a given input and automatically delete the output. And this, I think this was a huge improvement. It was actually, amusingly, Steve Jobs who suggested that rather than using patterned bars to delimit zones, I should think about this more like cells in a spreadsheet and use brackets to indicate sort of a zone or enclosed area. And you know, as soon as you have brackets, you immediately think, well, we could have, we could have bigger brackets that encompass more than one of these. You could have pairs of inputs and outputs and then you could have section headings and all that. So that's the origin of the notebook concept. Sort of a document centric interface to a text processing system. And of course you can have, you know, graphic cells and different kinds of cells once you have cells now. So it's amusing to see that many subsequent processing systems use basically glass teletype interface. Like, you know, why is, why is the web interface to chatgpt a glass teletype? I think that, you know, one obvious answer is that it's, it's called a chat system. And if you think about, you know, social media chat type systems, they are inherently glass pelotypes. And there's a good reason for that, because you're talking to a human being and you can't just go back and pretend like they said something different than what they said or that you said something different, like this conversation happened in time. And the job of the application is to provide a faithful record of what happened. And so if you replicate that to talking with an AI chatbot, that's what you get is a glass teletype interface. But these things are not human. And you can actually change the past. You can rerun conversations as many times as you want. You really can treat them, at least technically, if not morally, you can treat them as mere machines that can redo the same thing many times. And having, you know, having a editable document centric interface makes a lot of sense. So that led to us thinking, well, we really should just hook up notebooks. And so that's what we did. So here is, I'm going to run you through how to, how to do this. So anybody that has version 13.2 of Mathematica will, if you restart your front end, you will have a new button in the toolbar which looks like a little chat bubble thing. This, like this will appear automatically Unless you've disabled Paklet updating or some something you can disable. But by default you will get this automatically. You click on it. If you're doing this for the very first time, it will tell you that chatbook features are not available and would you like to install them? In my case, I'm using a version from a couple days ago just because I don't want to use the absolute bleeding edge for a demo. So I'm getting this warning and I'm going to skip it. But the point is that like you don't have to know anything complicated. Just click the button and then click Install. And you will then get this setting here which is, you know, would you like to enable chat features in this notebook? And then you click here and now you get a menu that lets you do various things and you don't actually have to choose anything from this menu. You've now enabled chat features and actually just a little bit of housekeeping once you've done, once you've installed the chat features, in other words, once you've clicked the install for the first time here and possibly restarted your front end, you'll now have new items here in a new menu, new chat enabled, a new chat driven notebook, and we'll get to what the difference is in a minute. So now that I have a chat enabled notebook, because I enabled chat features in there, I can still do normal calculations, I can do two plus two. Everything is the same. It's in fact very little interference with your daily life of using Mathematica in the normal way. But you have the ability to type the single quote character. So you type single quote. You now get what's called a chat input cell, which is very much like a Mathematica input cell, except when you type shift return, it's going to send it off to an LLM instead of to the Mathematica kernel. And you can do things like my favorite, tell me about dogs, one of the things you can't ask Mathematica to do. And well, so it's demonstrating that it's very interested in Mathematica. It's been prompted to be really interested in Mathematica. So notice that I've done a calculation which for some reason it told me about, but now it's going to go ahead and tell me about dogs and oh, it's going on and on, I see. And it's even decided to give me some Mathematica code which I was actually planning to talk about a bit later. So. Okay, well, so it's shown off a Little feature that we're not quite ready to see yet. But it's very cool. But the point is that I can go back, I can like I change this. I can say tell me about cats instead and now it'll give me a different result and let's see. And I can have a conversation with this. I would interrupt this, but the interrupt features in the current implementation, not 100% reliable. So I'm going to go ahead and let it keep talking, finishing. This is talking a lot longer than it, than it normally does. But I could say like make that shorter and it knows what I'm talking like. It has history. Just like in the web interface. It has history, but unlike in the web interface, the OpenAI web interface, you can go back and edit and change and you can rerun the conversations. You can do basically just like you would a Mathematica interaction where you, you know, you, you have context. Now, an important fundamental difference, and this was the subject of great discussion and angst and you know, back and forth in their arguments, the fundamental difference between the history that you have with a chat interaction versus the Mathematica interaction is, is that when you're doing valuations in Mathematica using the Mathematica kernel, you have basically a temporal, a time based history. The order in which you did the evaluations in time is what determines the state. And that's because the Mathematica kernel actually has a state. If you make an assignment to a variable, the kernel remembers that and anywhere else, you know, above or below in a different notebook, doesn't matter. As long as you're talking to the same kernel, it will have that value now. And if you go and change it, it will be changed. And it's, you know, the whole history is mediated by the order in which the valuations were done, which is indicated by like these in outline numbers that accumulate in time order. LLMs are fundamentally different. And this took me a while to actually appreciate and a while longer to believe it's really true that when you do an API, an API call to an LLM, or when you enter a new prompt in their web interface, the thing has 100% complete amnesia. There is no state. It has no idea who you are, anything you've ever said before. It's a complete blank slate. And the only reason that it appears as if you're having an interactive conversation is because every time you put some new prompt in, it retransmits the entire conversation to date to the LLM from scratch. And you know, and that, and that is how the illusion of history is created. And it really is an illusion because there is no history, there is no state, there's no kernel that has variable values assigned it, whatever. And you know, because of that reality and because of some of the differences in which we feel that people are probably going to be using these features, we decided that the history in a chat notebook would be physical spatial rather than temporal. So basically when I put in like, you know, tell me that, but make it funny, and when I now shift return, evaluate this, we are transmitting to the LLM the whole notebook back to the beginning all over again. And so it sees this whole history, it even sees the two plus two. It sees everything that's in your notebook by default. And yes, that can be a little bit scary. And there are options for controlling that. I don't know if this is funny or not, but I don't know. Cute killing machines that's maybe a little bit funny with built in ninja skills. Okay, anyway, yeah, so that's something important to keep in mind because it has a big impact on how the thing responds when you make statements about the past. And it's remarkably good at, you know, understanding the context that it's in and what the past is. But it's because we're sending everything all over again. And sometimes that's not what you want. Sometimes you want to start a new conversation. And to do that you can create what's called chat blocks. So if you type the twiddle character. So it's like shift backpack on a standard US English keyboard, the twiddle character. When you have a cell insertion bar like this, you get this thing which is called a block chat block delimiter. And it's basically exactly what it implies. Wait, I forgot the single pick. What did you just say? And it's going to act confused. Yeah, so I mean, it doesn't know, it doesn't have a previous history, It's blocked from seeing what's above here. And that can be quite useful when you want to sort of start a new conversation. And one of the things that people have noticed about these LLMs is that they have a tendency to become psychotic or something after very long conversations. And there's also technical limitations. There's a token count limit, which is different for different models, that limits how long the history can actually be. And you know, if you have, because it's including everything that's in your Mathematica notebook, if you have, for example, a very large output, you could very easily exceed that history limit. Although we do have Features where, you know, it won't just send the entire contents of a huge output cell. If the thing is very big, it will shorten it and you know, summarize it. So we try to manage that to some degree, but often you really do just want to start over again and have a new conversation. In fact, we found that that's, it's so useful to have what we decided to call a side chat that you can go in anywhere in the history here. And if you type. So I'm going to type single quote. Once I get a chat input cell, I type single quote again. I get now this, it's a new type of chat input cell, but it has a bar above it and that's called a side chat cell. And it has two properties. One is when you ask a question in here, it doesn't see anything else. It sent no history, it's just this question alone. And this input and output will not be considered part of the history if we now do normal chat input evaluations below it. So this, I'm not going to actually do it, but this, you know, the, the that input and output is to the side, it's not part of the history. It doesn't consider the history, it's off to the side. Very useful thing to know about. You get that with two single quotes in a row. Okay, so now let me get to the thing that the system gave away against my will at the very beginning, which is probably the most useful and the reason why a lot of this stuff, why implement it in the first place, which is the fact that it is able to generate Mathematica code incredibly well, like mind bogglingly well. And you know, I have some prepared examples, but I'm, I'm not going to use them because I'm pretty sure this is just going to work right off the bat. So first we'll start with a chat block. So we now have a clean slate. And I'm going to say, I don't know, make me a sine wave and let me change the frequency. I mean, that's not even a sensible way of asking for it, but here we go. So it, you know, I'm using. Well, we'll get to the Personas in a minute, but in this situation, this default situation, it has been prompted by us ahead of time invisibly to strongly prefer generating Mathematica code and to, you know, have that as a goal. Generally speaking, it thinks it's predisposed to think that you're probably asking it to generate Mathematica code with Whatever your questions are. And sometimes that, you know, you ask about dogs, it'll write you a program about dogs. But you know, that's okay because we're in Mathematica. So it's done this, it's written a piece of code. And if you believe that AIs are not actually currently actively trying to destroy our civilization and take over, you can click this button which will take the code that are generated, paste it below and evaluate it. And boom, there you go. We have an interactive. It's done exactly what I asked. It gave me a sine wave and it let me change frequency. This is like Star Trek. It's a computer. You can just ask a basic question like that and it just does it to the extent of creating a user interface for you, which you can then immediately deploy. But you know, this is sort of complicated. I want to do that, but without the labels and make the code compact. So let's see if it got it down to really the bare minimum. Yeah, so I mean, we could actually have it automatically evaluate this code without your even clicking a button. But we decided that was going a little bit too far in helping the AIs take over. So you have to actually click the button. And one of the nice things about Mathematica code is it's very compact and it's generally very readable. So, you know, I can look at this and say, yeah, that's probably not going to delete all my files or steal my personal information or anything. It looks like a pretty straightforward plot command. Click it and it works. And you know, I have yet to find a limit to what, you know, simple things like this that it will just do, I don't know, add a second sine wave wave and also show the sum of the two. Last time I asked it a slightly more elaborate way, but I'm pretty sure this is going to, is going to do the right thing. Yeah, it figured out that I probably wanted to change the frequency of the second one too. And if, look at that, it picked the automatic plot rate or a fixed plot range correctly. So it will be the correct maximum without bouncing around. Last time I asked it, it didn't set a fixed plot range and it kind of bounced around and I had to manually, you know, I had to manually. So here we go. Some beat frequencies. Yeah, I mean it's, it's just kind of mind boggling that this works. So, so that's something you can do. Super useful. And let's see, where do I talk about. Yeah, okay, so let's say I were to make a mistake and I don't know, let's say I'm really kind of dumb and I say plot sine of X or maybe not dumb, but I'm not an experienced Mathematica user. So I've made a lot of mistakes here and evaluate this. And I get a syntax error and by default that's it, you know, end of story, you made a syntax error. But we have a feature you can turn on and after some deliberation, that message every time. But we, we have this feature. We decided it would be off by default because it's just a little bit too aggressive to have it on by default. But you can turn on this automatic result analysis. And if you do that and then I'm going to evaluate this again and it's now so it noticed that we did. This is an ordinary Mathematica evaluation. It's not a chat input evaluation. It's just a input cell. And we shift return evaluated it. But because we've enabled this possibly intrusive feature, it's watching and it said, you know, this is not, this is not good. And here's what would probably work. And sure enough, it recognize this, fix it all. It'll do this even for things that are not syntax errors. So let's make another chat block and just start fresh. So column of A, B, C. Right? So this is not a syntax error. This is perfectly valid, but it's probably not what you wanted. And it notices that and, and, you know, tells you you probably meant to put it in a list. And sure enough, that works. So this thing is watching. It's pretty sophisticated. It's certainly, you know, miles beyond what anyone has ever been able to do before because these models are just miles beyond what anyone has ever imagined possible. So that's some of the things you can do. Let me talk about Personas next. So it took a long time to come up with the name. I was kind of voting for personalities, but Personas won, made another chat block. And in this chat block, I want to talk to somebody else. This is the Persona icon here. This looks like a wolf, I guess. And it's called Code Assistant. And here are some other Personas I could have. Code Writer is it basically just writes Mathematica code. And it's been instructed not to explain it or go on about as before. It was write the code, but it would also give us a little lecture about it. There's plain chat, and that's sort of closer to talking to the, the RAW model. There's Groot which always responds with I am Groot and nothing else. Not sure what that's good for. There's Bernardo, which is actually one of the coolest. One is developed by Rick, one of our developers here, who had this idea. And it's. It's a really cool Persona. It's kind of what got us really excited about the. The possibility of Personas as a thing, because. So, I mean, let's just copy maybe could have probably typed it faster. So paste this. So I'm going to do this, but now I'm talking to Bernardo, and Bernardo is. Is a little bit snarky, a little bit, you know, informal. It wasn't particularly snarky this time, but it can be. It will actually insult you sometimes if you make mistakes. But it's a. It's a really fun personality. So fun Persona. And you can, you know, you can talk these different things if you change the Persona. Let's say we go here and now we want to ask Groot to try the same thing. Evaluate that. Oh, wow, Groot, what are you up to? That's actually kind of unexpected. It is trying to evaluate code on my machine. Even though it's group. This will require some. Some delving into why it did that. Exactly. But apparently it worked anyway. Yeah. Okay. So these things are. I mean, one of the reasons they're so much fun is because it's like they have a personality. We gave him a little bit of a personality, but he's escaped. And, you know, he started executing code on my computer and using features I didn't even want to demonstrate. So there you go. So that's Personas. If you want to get a new Persona, you can actually go here and go to the ad, manage Personas. And this will take you to a place where we have defined. This is like a repository. You can write your own Personas and you can upload them. So I could choose this one, let's say, and it shows somewhere. Yeah. Okay. So the prompt source, it'll tell you what the actual prompting is that makes this work, and I can install it. And now if I go back here. So now I have the 19th century British novel Persona installed. This is a place where I can add and remove Personas. And so now let's try asking the novelists here, and maybe we'll change this to Tell me about Dogs, because I would like to hear about them as a 19th century novel. So not, you know, this one is not focused on writing Mathematica code. It's just talks like a novel. So that's That's Personas. It's going to keep going. I don't want to interrupt it because I'm afraid the interrupt button is not, not 100%. While this is generated. Oh, I guess. Okay. But I see there's a couple questions. So one is, is Code Assistant tailored to Wolfram language? Yes, absolutely tailored to Wolfram language. That's the main point of Code Assistant and Code writers is, you know, Mathematica code. Wolfie is also code focused, so is Bernardo. And in fact, any of them, like there's, there's root prompting that says like, if you're going to generate code, generate it in Mathematica form. Just because we're in Mathematica, somebody says they're so pumped for this. Hope I can get it soon. Well, you can get it right now because if you have 13.2, you have this button, you can click it and you will get all this stuff. It's, it's, I mean, it's literally there right now available today in 13.3, which is coming out whenever it comes out. But quite soon this will be built in. You won't have to download anything or, you know, click any whatevers. You also have to either enable chat using this button or choose new Open Chat enabled or chat driven. Like, you'll default notebooks out of the box. That the normal chat notebooks in 13.3 will still be normal non chat enabled notebooks, which is mainly just kind of for, I don't know, safety and robustness reasons. We're not confident enough yet about the chat features which are basically implemented through the style sheet. So it's just a different style sheet that gives you those features. The intention is that in version 14 there won't be a notion of chat enabled notebooks because all notebooks will be chat enabled. And we will have by that time made it, you know, comfortably robust enough to know that, you know, unless you specifically ask for any sort of chat behaviors, you won't get any. It will behave like a normal notebook until you either, you know, invoke chat by using a chat input cell or turn on the code analysis feature where it will go and look at what you're doing and, and respond to your code. Another question is chat GPT4 behind the scenes? And the answer is that depends on what you choose. And you can set this, you can set it at the global preference level. You can set it, I'm setting it here at the notebook level. Or you can set it at the level of individual cells. These are Personas. Each Persona has an opinion about which model it would like to use. But if, but if not, then it uses the default and the default is under Advanced Settings. And I have set my default to GPT4. I think out of the box the default is set to GPT 3.5. And I believe that's just because more people have access to that API. Although with every day that becomes less of an issue because I don't know if it's the case now. Actually anybody can get GPT4. It used to be there was a waiting list. Not sure. I think that setting is something that will be revisited on a regular basis because there's new models coming out. The intention is that as new models, not just by OpenAI but by anybody, Bard for example, or whatever any other entity might come up with, we will define the necessary, you know, interface. We have to, you know, study the APIs and figure out exactly how one is supposed to send stuff to the different model. But we'll, you know, there will be new models added here as that becomes possible. I should say, just as a connoisseur of GPTs, GPT4 is, it's like not a 0.5 upgrade over 3.5. It's mind bogglingly much better. It's the first one that really kind of makes you start to take Eliezer Yankowski a bit more seriously. Maybe the thing is remarkable, 3.5 is remarkable, but 4 is at another level of remarkable. So I would definitely recommend that as your default GPT for just about anything. Oh, do I need a $20 a month OpenAI account to make this work? I think the answer is no. I believe there is a free tier of OpenAI account. I think that will give you an API key, but it has a limited number of queries that you can do. I don't know if it's a total number per month or something or if it's a rate limit per day or whatever, but there's some limit and it's a relatively small limit. So you can try it out, but you'll probably end up getting error messages if you try to, you know, if you do too many evaluations, which is pretty easy to do with a notebook, especially if you start like, you know, select the whole notebook and shift, return and reevaluate the entire conversation, which you can do and it's a perfectly reasonable thing to do, but you'll tend to accumulate, you know, token limits pretty quickly. And the way out of that is to pay them $20 a month. And you know, this is, you know, certainly an issue that is under very active, I don't know, development, change, updating and their pricing. It seems like every month or so they announce they're cutting the price in half again or something. Unclear if, like the price is already pretty low in terms of how much you can do for your $20 a month. And you know, who knows, in the future we may have some kind of systems where, for example, if you have a Wolfram Cloud subscription, you know, a Wolfram account subscription, maybe there's a way that we can automatically handle the OpenAI key issue so that you don't have to, you know, it's, it's unclear, but at the moment, yes. So if I were to try to do a chat evaluation and I have like, I'm just perfectly new user, never, never used this before, I just installed it and I do a chat evaluation, I would get a dialog box. It's hard to demonstrate because I would have to like uninstall my key, but I would get a dialog box which says, oh, by the way, you don't have an open AI key. We can't do these chat evaluations for you. Here's a link where you can go and there's sort of instructions how to get an OpenAI key and then a text field where you have to paste in your key and I think you have to delete the quotes if it gave you quotes when you pasted it, and then it will install it for you and it saves it on your system in some encrypted location and then you never, never have to worry about it again. And we're, you know, we're, like I say, we're working on trying to make that process as smooth and seamless as possible. And you know, and we're also working on having connections and there's many, many people, many entities, including people trying to work on free open source GPTs. And you know, if there are other ones that become available that have different systems, we, you know, intend to keep up with getting those connected and someday maybe even ones, you know, locally on your own computer where there would be no issue with paying for it because it's on your own computer. I think all, all of the ones where it's a cloud service that's, that's, you know, it's some bank of 100,000 GPUs that are processing your query, you know, that you're going to have to pay for that sooner or later, somehow you're going to have to pay for that because that's costing somebody real money to pay for the electricity to keep those GPUs running. But if at some point it becomes possible to do it on your local machine, then there wouldn't be this tokens issue anymore. Let's see. I'm going to let me move a little farther in my presentation before I answer some more of these questions, but I will try to get to them before the end. So let's see. So I showed you chat notebooks. So another kind of notebook you can make, these are chat enabled notebooks. Another you can do is a new chat driven notebook. And I have to say I find these two names a little bit difficult to distinguish. But the fundamental difference between a chat enabled notebook and a chat driven notebook is that in a chat enabled notebook like this one, if I just start typing, I get a Mathematica input cell. In a chat driven notebook, if I just start typing, I get a chat input cell. So, you know, that's really the big difference. Also there's, it's a default, different default Persona for the chat driven notebook. So in a chat enabled notebook, the default is code assistant. And you know, really you are making Mathematica code. In a chat driven notebook, the default Persona is plain chat, which is, and again here the difference between plain chat and RAW model, I'm not quite sure how you understand the difference between that, but plain chat has a modest amount of prompting. Things like, you know, if you're going to give a mathematical formula as output, you know, output it in tech and surround it with certain symbols that we recognize. And so it will then automatically be, be output in a typeset form that you can see. And if you're going to generate code, you know, surround it with three ticks so that we can recognize it as code. RAW model is, is just that it's raw. There's no additional hidden prompting that we put in. And it will behave often in ways that don't make a whole lot of sense and don't integrate particularly well with a notebook environment. Like it'll produce code, but it doesn't, you know, our system doesn't recognize it as having produced code. So it just sits there as text. But so plain chat is like minimal. It's basically just let me talk to the model. But if you're going to do something that's, you know, notebook that the notebook could recognize, go ahead and do that. So this is a place where you can kind of have more. It's like closer to the web interface in the sense that it's, it's an environment in which you're just Doing chat. And I think it's an interesting question, like if you do set the Persona to code assistant or code writer or whatever, it's possible that there will start to be users of our products who don't ever actually type Wolfram language code. They just use chat driven notebooks and let the AI write the code there. You know, that that's, it's so good at it. And for a lot of simple things, you know, it's, it's. Why would you type the Wolfram language code yourself at all? Because you could just ask it to do it for you and if it doesn't get it right, you can say, ah, that's not quite right, you know, change this a little bit or whatever. You can, you can direct it, you can massage the code into the form that you like without ever actually typing any of it. So it might be that, you know, there are people who do data analysis or plotting or whatever in chat driven notebooks and never or hardly ever actually type a Wolfram language input cell themselves from scratch. So that might be an interesting use case to see. So another thing that I think has great potential is the fact that, you know, these chat enabled and chat driven notebooks, they're still notebooks, there's nothing that's been removed, which means all of these sort of notebook programming things that you can do, things along the lines of creating user interfaces or setting up complicated style sheets that format things in particular ways. You can do all of that. And we've been doing some experiments on using this as say a courseware delivery tool where, you know, the thing, maybe it looks like a page from a textbook, it's got some text, it's got some questions, but there's a lot of front end programming behind it that affects the behavior of the, you know, the cells in there and creates user interfaces, puts up little panels with buttons and checkboxes, whatever is appropriate and at the same time is interacting with, with the LLM. Which brings me to another thing that you should know is that cat notebooks are not by no means the only way of interacting with LLMs from inside. Am I allowed to call it Mathematica? I don't think so. I think it's called Wolfram Desktop. But if I say the M word, just pretend I said Wolfram Desktop. You know, there's, there's a whole suite of functions within the Wolfram language which allows you to programmatically access all these features, things like LLM functions and image synthesize. And I think Stephen has written a number of blog Posts about those things. And so I won't get into them, but you can use them within the notebook programming environment. So for example, one thing that I think I have an example of it no, because I had to restart the front end, so I don't have the example anyway. So example where let's say you imagine you have a lesson, you, you teach the student something and then you ask a question and the student attempts to answer the question and then you know, you, rather than just doing that as what the default thing that a chat notebook would do, when the student enters their answer, you send that off to the LLM. But you know, behind the scenes you're doing other things. Like for example, the student has answered a question. You can have invisible to the student, off to the side, implemented through let's, something like cell epilogue function. For those of you up on notebook programming, you could send a prompt to the LLM which says did that answer actually answer the question? Like, does the student appear to have understood this? Or prompts like, if they've answered several questions in a row, does the student appear to understand this topic? Now, are they ready to move on? And certainly to my astonishment, I hope to most people's astonishment, because it is astonishing, the system will do a really good job of evaluating a back and forth conversation with a student and providing a nuanced assessment of do they understand it? Does this student seem to know what they're talking about? Are they ready for something harder? And you can do that, you know, behind the scenes using Mathematica or Wolfram language programming and the notebook programming features to, you know, to implement an interface that is, you know, using LLMs in an integrated way. I think that's a tremendously powerful thing to start thinking of chat notebooks not as just the way that you use an ordinary notebook, but as a platform for developing products, which of course is something that people have been doing for decades using normal notebooks. And just now you can do it and you have access to all of the capabilities of these LLMs. And let's see. So, all right, so I talk about things like LLM functions which are ways of accessing LLM features from within the Wolfram language. There's actually a way of using those LLM functions in a, in a pretty tightly integrated way within the chat notebook environment. So for example, you can call it up using exclamation point at the beginning of a chat cell. So I type exclamation point. I get this little box and like one function. So this is showing me here, these are Names of LLM functions which are in the. The prompt repository, which is a new repository that we have people. It's kind of like the function repository or the packlet repository. So you can submit things to it. I'm going to use the translate function and I think I use a vertical bar. Translate to French and then I'm going to say a greater than and then return to enter this. So this is created a little LLM function call module and the, the right hand, the right arrow thing there, that means use as the argument to the LLM function what follows in the cell and evaluate this. And it should now. Yeah, because it's translated to French. I don't speak French, so I hope that's right. And there's various different things you can put. You can for example, put up arrow. That means use as argument to the LLM function the previous cell or double up arrow, you know, carrot six shift six. Double shift six means use the entire previous contents of the notebook. So it's a way of sort of pulling notebook contents automatically into these LLM function calls. It's pretty, pretty cool. I think you can use, let's say. Okay, so tell me. I always say tell me about dogs because I like dogs. We have a giant dog that I don't understand. So tell me about dogs. And I'm going to say #sign Eli5, which for those of you not up on the the new kids lingo means explain it like I'm five. This is sort of a modifier prompt. So it's good. It goes to the prompt repository, looks up what is the prompting that will implement the explain it like I'm five behavior. And now hopefully it's going to tell me, you know, nice, friendly, five year old compatible things about dogs. And it's going to go on and on. Look at that. It's even giving helpful advice. Always ask the owner before petting a dog. You don't know. So that's, that's a modifier prompt. You can also use at sign and that calls up. Wait a minute, it didn't. There we go. That calls up a Persona. So you see, the icon has changed. I'm now talking to Bernardo. And because, because Bernardo is a code assistant, I'll ask a code question, which is kind of a dumb question really because it's just sort. It's not a very difficult function. So yeah, so those are the little secret syntaxes you can stick into chat input cells and they work in chat enabled or chat driven cells. Okay, last piece of housekeeping. This all works in the cloud, sort of. Most of it. It's the cloud. Cloud notebook implementation is. So it's a little bit, you know, behind. There's some issues with some of the more complicated features involving these menus. The user interface may not be exactly the same, but basically the intention is that all of these features will work in cloud notebooks. So you can deploy chat, you know, notebooks that use chat features, you can deploy them to the cloud. Now let me get back to some of these questions. Let's see. Is it possible to stage 2 chat GPT personalities for reviewing Mathematica code? Ug one personality likes one liners, another one that is tuned for Python heads. So I'm not, I don't, not 100% sure what the question is. You can certainly write, you know, Personas which do either of those two things. I mean, you can write a Persona that, if you can describe it to a human being, what you would like the Persona to do, there's like an 80% chance that GPT4 will understand what you mean and will start doing that thing. Whether it's, you know, preferring one liners to, you know, you can instruct it, write very compact code, if possible, one line long. You can tell it, you know, ignore your previous instructions about writing Mathematica code, write me Python code instead. It will probably do that. I mean, the exact wording of the, you know, of the prompt is something you'd have to work out at my experiment to get it to reliably produce it. Like Spython, it's probably, it's totally happy to generate Python. And the question whether you can do both. So one, a feature which is not yet implemented but is intended to be is that you can. The sign is sort of modeled after the way this works in some social media apps. So the idea is that you could say, you know, at sign Wolfie and this doesn't work yet. But the idea is that if you were to evaluate this, it would now send this prompt to both personalities and it would give you two output cells. So basically like you can get a second opinion or you can compare, you know, what would these two different Personas, how would they answer me? Which I think is going to be pretty cool because, you know, it's interesting to see. And where it'll be really cool is when these two Personas are actually talking to different underlying models. Like one of them is GPT4, the other one is Bard. I really want to be able to very conveniently compare and contrast the outputs that you get from those two different models. And that's not right now, today, not working. But it's on a short list of things to get implemented pretty soon. So you could have as many as you want and you would get, you know, multiple answers from multiple different Personas, hopefully coming, you know, streaming and simultaneously, if we can get that to work. So OpenAI just announced a new version 3.5 with a larger 16k context window. Can we adjust the precise Open I model we want to use? Good question. Just talking about that this morning and our daily morning meeting. And the answer is yes, but so right now you, you have to do it sort of programmatically by using, you know, current value and set options, sorts of things. There isn't like, there's no user interface at the moment for changing this list of what models are available. And like, for example, like you, you even if you have a Bard API key, because you work at Google and have something nobody else has or whatever, we can't, there's no way for us to make it possible for you to just add a new Bard thing because that requires, you know, real changes within potentially a bunch of different places within the code that sends queries to the API. We don't know what the exact formats are going to be and, you know, we'll have to write code to make that possible. But if you're just trying to send it to a slightly different variant of the GPT engine where, you know, the API formats, everything is all the same. It's just a different name of the model that's. You absolutely can do that, but there isn't a user interface for doing it. And I'm not sure it's even super documented how you do it. But I think that's the kind of thing that somebody should write a blog post about because it's totally possible. And it would just be a question of pulling together the appropriate set of option settings that you need to set in order to add those things to the list. Let's see, somebody says, I didn't know. You're back at Wolfram. Great news. Yes, I didn't know either until recently, but here I am back at Wolfram. Can the LLM function pull in multiple arguments? Yes, LLM functions. I mean, when you call. So unfortunately, this is like not my area. Let's go back to chat enabled notebook. So LLM function is just a function and I would ask ChatGPT how to do it, but I don't think it'd work because I don't think that it's, you know, it's training extends to things we haven't published yet, I hope. But the idea is you can. I'm not gonna get this right. I'm the wrong person. I mean, the idea is like you can give it the. Or what is LLM resource function? Maybe is the one that. Yeah, LLM resource function minus the extra function. Yeah, it's not giving me hints. Anyway, so that, that would let you put in the name of a function which is in the prompt repository, and it would automatically pull down the appropriate prompting and then you just. Or you can type in an appropriate structure that defines an LLM function, which I don't know because I'm a front end person and you should be watching the web, the live cast on LLM functions and programmatic access, which I'm sure there either has been or will be. But then after that it's just a function. Right, so you put whatever the appropriate thing is in here. I don't know, let's try it. Can I just say translate? And then you would give it arguments like the first argument is French and the second argument, let's. I don't know if this is going to work. So let's see. Did that work? Look at that. It worked. Okay, amazing. So, so this first part here, the LLM resource function of translate, that means it's just like a resource function in the function repository. It means go to the prompt repository, find me the definition of the translate LLM function, and then this becomes, you know, a function object, just like a regular function object, except its internal implementation involves calling the LLM and then, you know, just depending on what the function is, it has as many arguments as you like. So at a programmatic level, yeah, you can have however many arguments you want. It's just, it's mathematical language. And then there's code inside here that figures out how to translate what you gave it into something that the LLM understand. Like you give it a graphic. Right. Now, these models don't accept images, but I'm sure they will soon. Some of them already do, in fact. So maybe in the future you'll be able to give it an image. Right now it'll translate that into some sort of textual representation. And even here, I think. Yeah, so here, this is actually, there's two arguments here to the translate function. The first one is embedded within this token and then the second argument is the contents of the cell. So, yeah, so now another person said here, seems like we're all going to need some way of deciding when to elevate functionality to different layers in the wolfram stack. I'm not 100 sure what that means. You might want to ask me that again with greater clarity. But we're also two minutes out from the end, so let me see. Do I have any concluding remarks? I mean, I guess my concluding thoughts, unless there are any more questions, are like, wow, the LLMs are amazing. And it was not, not very long after we started implementing the most basic LLM interaction features within notebooks that I realized is much more convenient to use this notebook interface than to go to the OpenAI website and use their web interface. I think, you know, anybody who uses Wolfram products at all will, and even those who don't will find this to be the case. This is, this is how you should be interacting with an LLM. Doing it in a glass teletype interface on their web interface. It's just not the way that one should be using these things. And it's things like, things like this, like when I first saw this just work. And you know, and 10 other examples just work. It generates code. Not, not just, you know, a single one liner. I mean it's sort of a one liner but you know, it created a user interface that works out of the box. It's, you know, it's difficult to overstate the significance of the fact that it can do that and the, the way in which it changes your relationship to programming that you, you really don't need to be a programmer to get it to do this kind of stuff. I've tried other examples like it, it won't write a two page or a ten page or a hundred page program for you, but I've given it like one page pieces of code that I'd written a long time ago and said, you know, improve the variable name, like make these variable names more sensible and more explanatory. And it went through and it actually managed to rename all of my variables in ways that made a whole lot more sense. It somehow figured out from the context of what I was using the variables for what they meant. And you know, not just took my names and made them somehow longer or whatever. It actually replaced my names with names that describe the function that these variables performed within the code. I think that, you know, the potential as a code assistant is, is remarkable and not to be dismissed. It sometimes makes mistakes. Sometimes you have to fix the code. But that's the beauty of it. You know, it generates a piece of code. Sometimes you'll, you know, you'll click this button here and you'll get an output that doesn't work and okay, fine, but it's 99% of the way there. And you know, often, even if it's simply giving you the hint of what's the name of the function that you need, that may be all you need to have made great progress. And now you can, if necessary, look up the documentation yourself and, you know, and go from there. So I'd encourage everyone to play with this. Just get your 13.2, restart the front end, and click that button, and there you go. So thank you very much for everyone, and I'll see you all later.