Getting to Know the Wolfram GPT

Hello, everyone. Welcome back to another live stream. Today we're going to be talking about the Wolfram Custom GPT. So you might have heard of custom GPTs and be curious about them. Today we're going to show you some examples from the Wolfram Custom GPT. You should have a link in the chat already that you can reach this page and take a look at some examples that we've created. A little gallery, other resources about AI tools that you might be interested to use in the Wolfram ecosystem. So today I'm joined by Michael Trott, the executive Chief Scientist of WolframAlpha. Also Alan Joyce, the director of content development for socioeconomic content for WolframAlpha. And they're going to show some awesome examples of what you can do with the Wolfram GPT. So without further ado, I'll stop sharing that screen right now. And I believe Alan's going to lead in with some examples of what are the sorts of things that you can do using the custom GPT.

Sure. Does everybody have my screen? Okay, so I guess first we wanted to just point out to people that if, if you have access to ChatGPT plus for right now, that's one of the prerequisites for being able to access the Wolfram GPT. You can go to this link and maybe John, you can share links as they come up in the the chat, maybe for people. And right now OpenAI is sharing some featured GPTs here. Certainly you can, you can search for the Wolfram will show up at the top of the list. Click on that. And if you've got Chat GPT plus, this will bring it up. If you're familiar with the ChatGPT plugins that were introduced earlier in the year, for the most part, you know, this is the direction that OpenAI is going with having access to other services within ChatGPT. So for the most part, the kind of functionality that you have access to here should be roughly similar. Michael will cover some of the key differences in the user experience, but for the most part, you know, anything that you can answers that you can get from WolframAlpha or anything that you can do for the most part, with writing Wolfram language code and evaluating it in the Wolfram Cloud, you can now do through the, the Wolfram GPT. So just some simple examples, you know, just kind of basic plotting functionality. The GPT knows, okay, I can write some Wolfram language code for this. It's going to send it to the Wolfram Cloud, get a result back and show you that image. You can refine those results. Sorry, one second. Not having luck with being able to scroll here, you know, you get a result back, you can refine that chat GPT will refine its code, send something off, evaluate it in the cloud and you get an image back within that chat session. Part of the prompting for. Sorry, part of the prompting for the. Sorry. Part of the prompting for the Wolfram GPT is trying to give it some advice about when to call WolframAlpha and when to write code for the Wolfram language. So for a lot of kind of simple queries, the kinds of things that WolframAlpha is sort of the easiest path to get detailed results and images and all sorts of things, it'll just call out to WolframAlpha and bring in those results. So for something like when's the next solar eclipse visibles from Champaign, sending that query out, bringing back, you know, a lot of the same information that you get from the full WolframAlpha result. And also generally the GPT is gonna include links out to the actual WolframAlpha results. You can click through and see, you know, some of the images. Sometimes it will show the images in line or you can just ask it to show images if you want, or just click out and it'll go out to WolframAlpha and show you those full results. For some other things, like here's something asking for dollar euro conversion rates and this is the kind of thing that, you know, chatgpt on its own doesn't have access to sort of up to the minute data in different domains. And so this is one of those areas where, you know, you can definitely, you know, get better results, get access to more current live information by going through wolframapha and the Wolfram language. And this is kind of showing that, you know, there are some cases where here's a query where it decided, I'm just going to call WolframAlpha. It gets kind of a summary of the data. You could also follow up on that and say, you know, can you show me a plot? And in this case it's saying, well, I'm going to go and I'm actually going to write some Wolfram language code for this. And so that's something that then you could refine this plot, extend that time series, ask it to, you know, change the visual look of that and it'll write some code for you there. So, you know, basically the same kinds of experiences that you had through the ChatGPT plugin. But again, having access to all of.

The.

Data in the Wolfram knowledge Base and all of the kind of functionality of the Wolfram language being plugged right into the Wolfram GPT. Michael, maybe you were going to talk a little bit about some of the differences between what people might be used to with plugins and how things appear now with the Wolfram GPT.

And the good news is that it's primarily not going to be a lot of changes that you'll notice. But as Michael is going to show, there are a couple of things to be aware of if you're used to one as opposed to the other.

Let me share my screen. Good. Something like smaller, three roots. Yeah. Okay, so let's look at some relatively simple example. Say, let's take this one. Say, make a plot of pairs, base exponent of base, where exponent of base is the smallest power of 2, base minus 1. These are all variables appears somewhere in 2 to the exponent of base. And then make me a plot how this looks like it's not to be expected that basically every number appears in every exponentiation. So there are probably gaps. So let's look at, here's one that I did a while ago with the old plugin. The plugin has not aged, but okay, good. And you see it made here a call and here it showed actually what it did. It wrote this piece of wire from language function, which is a little bit unreadable because of the new line characters. But if you would copy it out to two expressions comma, standard form, it will look pretty nice. And then it evaluated the last part of this, which is a list plot. The list plot result of a list plot is a graphics. Because the Wolfram language native graphics is not so useful for ChatGPT. We also make image and we tell ChatGDP you find the image here. And then ChatGPT has inlined it and here it is. And now let's do the same with the waveform plugin. No, sorry, not with the plugin, with the new Wolfram GPT. And it will look pretty similar. Okay, so this is a problem that somehow appeared literally in the last 24 hours that some saved chats lose their image. So there's this very similar image which should be here, which isn't here. But you see here is also a talk to ChatGPT from Cloud.com, but you see here one difference. It also formatted it at one line. It's a little bit harder to read, but you can still copy it out. But the big difference to earlier is you don't now get the output anymore, which might look Streamlined. I personally don't like it so much because it makes it a little bit harder to differentiate when the chatgpt makes something up on their own or, or believe it knows it or when did really the Wolfram Cloud or Wolfram Alpha send something back. But otherwise, functionality wise, the two are quite similar. And if you use it say for code writing and had it early or write some code that would make it in a black box and I think we discussed code writing in 10 minutes or something like this, you would not see any substantial difference between the plugin version and the new Wolfram custom gdp.

Yeah, so. And again, you know, there aren't too many cases when you're using it for various things that you'll see some egregious oh, it got back one response and then just ignored it and made up something totally different. In principle these things are possible, but you know, in practice, luckily I don't think happens too much. But yeah, that is probably the biggest difference to be aware of. So another thing that people I'm sure are curious about, since we're talking about how these sorts of things work, is what, what actually happens when something like ChatGPT is making a tool call? How does this work to have the ability for a model to tag in and call various APIs and things and add to the functionality that large language models have by themselves?

And I assume I should share again.

Yeah, let's.

Okay. Okay. So LLMs are really popular since pretty much a year and maybe a year and a month and they are used everywhere and there was a lot of excitement what they will be able to do and that still is. And we still don't know quite where their boundaries are. But there were, I think two big insights which have become clear over the last year and some people will say they knew it already a year ago. Number one is you can only get really good results out of an LLM if you prompt it properly. And I think we later on a little bit to discuss this. And the other one is an LLM might know a lot, but it definitely A doesn't know everything and B it is relatively poor at computation. Like if you take two 10 digit number and multiply them in an LLM natively you both probably get a 20 digit number out and probably the first two digits are right and the last two digits are right because basically it's not some kind of order of magnitude. It knows 10 to the 10 times 10 to the 10 is in the order of 10 to the 20 so it will not get 4.7 out so it has a certain feeling developed. But the middle digit of this high probability4 and the lesson to be learned from this is that LLMs become vastly more powerful if they have tools at their disposal. So if they can call knowledge bases for data, if they can call Wolfram language for computations or other things. And there was a recent preprint just last week, 24, no more from Arxiv and I really like the title if LLM is the wizard then code is the one. A survey on how code in power slogan language models much more than ever before. And I think I totally agree with this because it really allows to do pretty amazing things because the LLM will understand natural language, it will write some code, call some code, and then the even more powerful thing is the LLM can look what comes back and if it's not what you would naturally expect, then the LLM can can go back and make refinements. And this I feel like is really where the whole circle shines. So here's a little bit of a breakdown schematic what happens. So there's a user, there's OpenAI ChatGPT and there's the Wolfram plugin which is Wolfram Alpha, Wolfram language in the cloud and potentially other things. At this level there is already the training happened to the LLM and there's probably some safety prompt from OpenAI saying don't tell people how to make bombs and other bad things. And then we come into play, then we say when should the LLM actually call us? What kind of things can we do? And if it call us, how should I call us? Like forward formalpha, write a short pointed question, not too much fuss in it and expect a quantitative answer. Or from language, do write some code that we can execute, make sure you don't make mistakes in the code, make sure you don't make mistakes in how you call the API. And then comes the user in and the user now starts fighting their prompt. And for many cases I think it is really recommended that the user first follow some of this general prompting techniques. And the number one that over the last year was emphasized in literally thousands of papers is chain of thought. Basically tell the LLM, do not hurry, take your time, think through, reflect. If you have a question, ask me and so on and so on. And I was actually just today was an archive blueprint that I think was the first one that I saw that actually tried to quantify this and basically saying oh, they did a good job, I need more steps. So let it go back, let it make even more steps and the quality of the result and the probability of correctness is strictly linear to the number of steps the LLM takes. So really encourage the LLM to do a lot, even if maybe it costs you 2 cents more and a minute more waiting, it's often really worse. So now you have this squared away and the LLM knows what you Then you tell it the IQ problem. You want to sum this here. Yes. You want to make it an interactive demonstration or whatever is your task. And then the LLM goes to work. And for simple things like some of the examples Ellen showed, it tries first a Wolfram Alpha call. Obviously if you first say write Wolfram language code, it will not call WolframAlpha. But in many cases it first starts with this. It gets a result back. If result is fine, the LLM rephrases it, makes a nice sentence out of it and sends it back to the user. And then maybe it continues because the task was more complex and you wanted more things. And this time, purely hypothetical, the Wolfram alphacore fails either because we didn't have the data or we didn't understand, or the LLM traced it strangely or whatever. Not always report back a I encountered a problem. Let me rephrase this, let me try this. And so it goes a couple of times back and forth. And then it might decide now this time I might use Wolfram language, make a call to this, get the result back and so on. And so sometimes you see a 1012 time going back and forth. And ChatGPT will keep you in the loop with various degrees, sometimes with great detail. Sometimes it will just do it, do it, do it. Then it becomes a little bit boring to watch. But if it keeps you in the loop, I find it quite fascinating to follow its thought process process because sometimes, and sometimes it goes totally off. Like I was an example on Mass Overflow, a nice binomial sum and with a little bit of guidance it can actually do it. But if it sees somehow the word binomial and asymptotics, oh, I should use formula if it's of relevance or not. It's somehow so deeply ingrained. My savior. Which it isn't. So it's kind of funny to watch this interaction and then you get your answer back and then often you do a follow up question because you have a related question or there's something more to be investigated. And so until.

Yeah, so we have an audience question actually, which is related to getting specific modifications to some graphic that's come back. We can talk a little bit about prompting techniques. We're actually planning to in a little bit more detail so quickly before that though, we also wanted to touch on. So what sorts of tools is this Wolfram GPT actually using? Because, you know, not only is there a GPT that's been built using these tools, but just having APIs for knowledge bases and being able to, you know, quickly develop things and have specialized APIs is also something that people would be quite interested in. And so we'll briefly talk about that and then I think we're going to talk more about some prompting techniques and also get to this audience question about. Okay, how can I specifically tweak some of this output?

Okay, so I, I think I'll jump over to me again. Have you got my screen again?

Looks good.

Okay, so what I think we just wanted to clarify, and this is, this is not really different from the basic architecture of the Wolfram plugin, but for people who weren't familiar or didn't have access to the plugins, or just curious about how it's all operating under the hood. The, the core functionality of the Wolfram GPT is built on a couple of custom APIs that are for the most part written in Wolfram language and deployed to the Wolfram cloud. There is the WolframAlpha API for LLMs, and the Wolfram Alpha has a number of different APIs that are available to partners to license and use. The API for LLMs was something we started developing about a year ago, initially for the plugin, realizing that the standard kind of super dense XML structure of the regular wolframal API was not really what an LLM wanted to see. We wanted to see mostly sort of a more basic structured view of all the Data in a WolframAlpha output, links to any of the images so that they could be run in line with Markdown. And so the API for LLMs is something that's really meant to be read by an LLM and not by a human, you know. Right. It's not an LLM running on our side. It's trying to feed things to the LLM in a way that it can most efficiently consume and display that data. And there's also a custom Wolfram Language API. And so this, as is sort of explained here, the Wolfram Language API can evaluate, you know, it just takes any chunks of valid Wolfram language code, evaluates them in the cloud, and then returns a response to the LLM. And initially this was really just returning results. And then we sort of discovered over the past year there were a number of other things that we could do to get a better result out of the LLM, even though it might take another sort of step with the LLM. So we're, if there's a simple, there's a result, if there's a visual result, it gets rasterized and returned with a link so that it can be run in line. If it's just data, if it's text, if it's numbers that gets returned directly. But then there are other things, like if the result of some computation is a very large list or data set or some more complex structure, we'll return some other fields and kind of say, you know, these are the dimensions of this data set. Because certainly if it's a lot of data, we've got a token limit for any kind of exchange with the LLM. So we have to keep the payload pretty small, going back and forth. So we're sending back. These are the dimensions of the data set. Here's the first row, here's the last row. Giving the LLM a sense of this is what you have to work with, even though you can't see it all at once, this is enough that you can sort of ask questions and maybe write some more code to dig in and give the user what they want. That's ended up being really effective. I think when you're, especially when you're working with data and larger chunks of data, it makes it easier to do that. There's also some other things, I mean, we've seen, and Michael can speak more to this, certainly over the course of the past year. What are the kinds of common errors that ChatGPT makes when it's trying to write Wolfram language code. And so there are some things that we're doing on our side to automatically correct common errors that we make. And we don't do that silently. We'll send back a result, but also say, this is what we altered. There was something that wasn't quite right. This is how you should do it. And so at least within that Chat session, if ChatGPT is continuing and writing that code again, it's not going to make that mistake again. So within, within that cycle, it's going to learn a little bit about how to, how to keep improving that code and other things like dealing with entities or properties and using interpreter in the, in the Wolfram language, just sending back sort of tips about how to write better code in some areas. And Michael, I don't know if you had any other specific cases that have come up over the past year that have Been kind of interesting with LLM written code.

Yeah, so there are a couple like some are. So always remember how the LLM works. It read a large amount of text and synthesize some probability distribution over some gargantuous dimensionless space of possible sentences and then you give it a new fragment, it calculates one of the high probability continuations. And there are some things which it has so deeply ingrained that even if Allen tells the LLM in the prompt not to do will not always obey. A very typical example is you make any computation, say where you want to calculate some electric field and it will use a capital E for the electric field. Now in Wolfram language the capital E is reserved to Euler's constant and we cannot Even not for ChatGPT make an exception and make this the elect. So it will make this wrong and you have to say hey, you forgot, remember you can't do this. And then oh sorry, I will fix this. And then it writes E field electric E or something like this and another one and this is probably asking for 2023 for too much, but maybe in two years it's different is while ChatGPT has learned to write pretty decent Wolfram language code syntact and also semantically, what it has not grasped yet is the whole story of evaluation order and even most mathematical from language users don't need it so much. But there are cases where you need it and a not uncommon scenario is for instance, the following people want a plot of a function and the derivative and sometimes you get immediately what you want, but sometimes you don't get what you want because the following happens. So ChatGPT sits down, writes a couple lines of wolf from language code. The first line is something like f of x equal and then your function sine of x times cos of 3x and so on, whatever. The next line is f prime, the prime really written out prim equal to colon equal F prime of x, which is not bad, but we should start thinking and the third line is plot of list F of x F prime of x x between 0 and 1 and then it fails. So why does it fail? Because the plot function has a hold all attribute and this is where slowly the knowledge of ChatGPT at the current status ends and it doesn't evaluate its arguments so it takes the numeric value of x. That's a reddavan plugs it in in the f prime of x the f prime of x was the function that differentiate with respect to argument. But after plugging in it's a number and you can't differentiate with respect to a number. So D is unhappy and say, hey, can't do this, don't know what this means, and then this whole thing falls down. Often ChatGPT recovers and said, oh, hey, I see, I made a mistake. I should. And evaluate around the first argument of plot. That's what it most frequently does. The other possibility that I have seen very rarely chatgpt doing is it could go back and change the satellite in the derivative definition to a pure set, but it rarely does this. It's not its favorite mode.

Yeah, no. Another one that I was just going to mention was that sometimes it will use print when you don't need to use print. Right? Because in many langu, if you want to output something, print is a fairly natural way to do it. But especially if you're working in a notebook, if you just want the output of something, you just run the thing and when it evaluates, you get the output. But if you're doing something where strictly speaking, the output of print is null, and then if it's run in a notebook, it prints the thing in the next line. But if you're doing programmatically something, you might get a null out of it using print when it didn't need to use print, that's another thing that can sometimes happen that people should keep in mind. So those are some, you know, coding mistakes that, you know, deeply ingrained from whatever training sets it has that, that be aware, you know, it doesn't always obey instructions to the contrary to not do those things. And so just when you're, when you're working with it, that can sometimes be the cause of getting results you're not expecting. And especially if you're wanting to use it as some kind of code assistant, just be aware of those sorts of little, little details.

And another nearly funny one is maybe in 10% of the cases, it thinks, oh, I need a variable, so I have to introduce the variable. So it writes X equals symbol of string X, which will not do any harm, but it will not do any good. And sometimes it prepares a dozen of the variables that it wants to use later where there really is no need for it, obviously. Yes.

So one of the things that, you know, are interesting about these large language models, while they're on the topic of APIs and things as well, is. So Michael was referencing a paper where, you know, the length of number of steps is correlated with how well it actually does in, you know, carrying out some plan, executing it. There's another paper actually that Michael, you sent me where there's an interesting plot that showed with the number of options available to a LLM, it can get confused by the greater number of options. And so knowing. So, you know, the Wolfram GPT has the power to be able to do all sorts of things. But Alan was also going to talk about how it's possible to, if there's a very specific thing you want it to do, in other words, for your specific application, when you're wanting to, let's say, write a quick API using Wolfram language that, you know, calling any kind of, let's say GEO functionality is always the wrong thing because this is not a GEO application, this is specifically some image processing application. You know, let's maybe talk a little bit about more about A, the sorts of tools that we make available to the Wolfram custom GPT and then B, how people can, if they're interested in a very specific API, do some things related to that.

Oh, sure. So, you know, in addition to these couple of APIs, you know, the core, these are really the core functionality of the Wolfram GPT. So the Wolfram Alpha and the Wolfram language portions. One of the other things that we, we discovered early on was there are certain kinds of things that ChatGPT was really not great at writing bespoke code for and that just would waste a lot of time and could be done very simply. So things like dealing with the knowledge base in the Wolfram language, finding representations of entities or properties or, you know, finding a, a Wolfram language expression or an entity value call for doing certain kinds of implicit entity class lookups, you know, doing Find me all the buildings with heights over some, some number of meters in New York City. It would try to write those kinds of things, but often it would make up standard names of entities or entity classes or entity types. So what we did was really just introduce a small number of very kind of single purpose again, a couple lines of Wolfram language code and deployed in the Cloud Cloud APIs and then added to the prompting for the Wolfram GPT. If you need to find the right standard name for an entity, send the type of thing and the name of the thing to the Find Entity API. If you want to find Wolfram language representation for a more kind of complicated natural language input like tallest buildings in New York City, just send a simple natural language input to the Get Semantic Interpretation API. And so it was a thing that was really great because it led us kind of, as Michael said before, that, you know, LLMs are very good at tool use. And so there were some cases where we realized, rather than asking it to become an expert in this one Wolfram language tool, when they're sort of very specific tasks that come up a lot and could be done a little bit more efficiently by just having a separate tool for the LLM to use, it does very well with using those things and using them at the right times. And so that's also something that, I mean, we might as well kind of touch on this now. We. Some of the other things that are possible, you know, if you're, if you're a Wolfram language user and you want to write your own custom GPT, you can, you can augment some of the things that are already in the Wolfram language by writing some of your own submissions for the Wolfram function repository. And in fact, we directly use some things that are in the function repository as part of the Wolfram GPT. Like we have this nutrition report function that was written by people on the food nutrition team. And this is just kind of a really simplified, efficient way of saying, here are a bunch of ingredients, you know, give me a nutritional report for this thing. So we added some instructions to the core prompt for the Wolfram GPT and said, if somebody is asking about this specific topic, use this resource function. And it works really nicely. Like, you know, you ask for some list of foods and the Wolfram GPT will say, you know what, I should use this resource function. I'm going to do a little bit of basic conversion here and then I'm literally just going to make a call in the cloud. The Wolfram language code is just going to be resource function of that and send those things and then it gets back all this useful data. And you could go further with that and do more computation with the data that comes back. But that's another way that for us, it was great in getting better results in very specific kind of silos of content. And it's a really easy way for other people who, you know, you deploy a Wolfram language API. I had another example just, just to show this is completely stupid and unnecessary, but a custom GPT that just makes word clouds out of whatever text you send to it. And this is literally on the back end. It is, it's a single cloud API that is one line of Wolfram language code that's basically just saying word cloud of whatever, the input string. And I even asked Chat GPT to write the OpenAI spec and then pasted it in here and it worked just fine. But, you know, it's a Great thing that you've got access to, you know, these kinds of tools. You have an idea for an API, you can prototype it quickly, deploy it to the cloud, go in, create a new GPT, plug in that endpoint, and, you know, presto, it'll. It'll make those calls out to your API, get the results in whatever form you want them in. But it's just a lot of flexibility, I think, for people who are just getting started with trying to build their own custom GPTs, and you want to do a little bit more than just, you know, custom prompts and have that be your whole custom GPT. You want some other functionality, you want some computation, you want, you know, access to other kinds of data. Using kind of the Wolfram Cloud and these tools is a great way to kind of quickly add a lot of really sophisticated functionality there.

Yeah. And just taking that workflow from the top one more time, it's just so that people sort of take. Take this message home, is that, you know, for these very specialized things, even though, you know, an LLM could in principle write code to get, you know, the. The code that you wrote for the. Or, excuse me, the food team wrote for that specialized nutritional information function, that's all written in code that the LLM in principle could have written. But why trust the LLM to write that code when you can just do it and then provide that as an endpoint for the LLM to use? And then this greatly increases its ability to reliably do the part that it can do of interacting with people and deciding what tool is good to use. And then the actual code can be something that you've checked ahead of time. Yes, this does what I want. Want it to do.

Yep.

Awesome. And so as part of building GPTs, prompting is something that we've discussed a little bit. So there are various prompting techniques that, you know, Michael touched on a little bit. One of them is chain of thought. And one of the things which is maybe a little bit frustrating to some about the current difference between a plugin and the GPT is that the one little piece about what the custom GPT is getting back from the API is a part of the chain of thought that you can't necessarily, as a human check because you're only seeing what the input was and not directly what the output was. But when it comes to examples that illustrate some of these prompting techniques, I know that Michael has great library of things that have been. He's been working on that can show a lot of very interesting techniques.

Okay, let me check in my screens. Okay, so basically over the last year there has been multiple papers per week that discuss pub dating techniques and it's basically become nearly a whole science. To keep up with them is nearly hopeless. And it's not even clear which one are really the one that of interest because you can't literally have 100 different things in the beginning of your prompt. Just laying out techniques of the art is even things like you take one LLM that writes the prompts, modify it, you have a second LLM that carries it out, you have a third LLM that evaluate what was done and say go back, make a change in the prompt. But of course then this becomes very time consuming and that's not really for a daily use right now the right strategy. So just a couple words about prompt strategies. The quote that had earlier about tools, just as a key unlocks the door, a prompt unlocks what an LLM can do. And I think that's really true. It's really pretty amazing what a good prompt can do. And really don't take a shortcut in writing your prompt. Take all the time you need. Write it very, very detailed and the more detailed and the longer the better. A couple words as already said, that's very important. And give it a structured prompt. We have a couple of lines of white spaces between this the problem. This is what you want to do. This is the result I want. This is the shape of the result I want. So tell it in great detail and there's no time to go today to all the about this. Just you can quickly look through if you see one that might be of interest to you. One of the ones that I personally like is telling the LLM to make test cases to check if the result is correct. For instance by if it's just a simple problem without code writing. If you are done and you have a result, take the result, go back to the beginning, plug it in and say you had a set patience today. Suddenly all become true. For more complicated problems. You want to tell it to not just think about it, but think about how you could split it up. You might want to ask it to actually not immediately start working on the problem, but take a break, think on similar problems or switch split it up into sub problems or come up with three different ways to approach the problem and then reflect which way to go. And sometimes you can tell it, come up with five ways to go and then let me decide which path to go and what else do we have to say about this? And for code writing, that's code writing. Is similar and but also has their very own things. So one thing that often makes code writing better are two techniques. One is either tell it to write very commented code and ChatGPT do a remarkable job in writing commented code. That's really quite impressive. Okay, I wouldn't use the same capitalization, but that's a minor dealer. And the other one is tell it to use descriptive variable names and it will do an excellent job of write code where you can literally read off from the variable names what it wants to do. So that's really quite nice. If for more complicated one, you can also tell it to think about the complexity of approach like you have a little problem. It doesn't matter if it's exponential in complexity if you just have five numbers to deal with. But for more complicated one, you want to have it reflected. And as convenient as it might be, you never, never ever tell it. Just give me the answer. Yes, it will be fast, but it will be formed with a good chance. And there are more minor details. As strange as it sounds, it pays attention to capitalization, it pays attention to emojis, it pays attention to saying you get a reward and it's really useful for me. And such things you can also say to improve the quality of what you get. So let's look at one concrete example maybe.

Yeah, well, Michael's bringing out this concrete example. I should also say so because this is such a large catalog of techniques that are being constantly developed in the literature. If people have questions about those, do feel free to reach out. @academic innovationolfram.com we actually give, you know, sequence of presentations about, you know, catching up with these sorts of techniques. If you're someone who is, you know, not, you know, deep in the literature on that sort of thing and wanting to just use these things in a practical way, we try to help you bridge that gap. So yeah, Michael has some, some examples here so that concretely illustrate some of these things.

So here's one thing. It's a simple problem. Find the smallest value for a natural number A such that the following cubic equation has three real roots. And now I say you must approach this systematically. I want to see all intermediate results because sometimes it keeps them a little bit hidden. And then in a complicated problem, I'm a little bit lost what we are doing right now. And another one, if it uses code, I want to see it this one, it's not always obedient. And here is its answer. So it starts repeating the problem. Then it makes a nice outline. This is one of the nicest things when you ask it to do it step by step, not only does it the step, it first outlines the steps before it goes into executing the steps, which is very nice because if something goes wrong, you think, eh, that's not quite right. You could interrupt here and say, hey, you forgot something. So then it starts with step one and very systematically it says, I have a generic cubic and I have this concrete cubic. These are the associations between the generic coefficients and the concrete coefficients. Then it makes a call to discriminant. It calculates the discriminant, gets this discriminant. Then it says, okay, I have real roots if the discriminant of a cubic is positive. So it demands this. Then it makes a call solve. And say might be not ideal for solve. Let me use reduce, which is a very clever idea because you might get a much more detailed solution. So it makes it the second call to reduce. Now, of course, we don't know what it got, we have to rerun it. But it tells us it got 27 4. And because we explicitly asked in the beginning beginning for a natural number. So we have one more step, which is easy, one we have to take ceiling of this and it gets seven. And originally there was also an image here, but we can go to the notebook and look. So that's the discriminant. This is the call to reduce and say to really believe that is true. We make a contour plot of the zeros and we see exactly here at 27 over, we start having in every slice three. And then of course we could make it manipulate where we really see three real roots in a log plot as deep dips. So that's, I think, a nice example, a simple example where telling it to approach it systematically, step by step, show me intermediate result really makes it very nice to follow and has a higher chance to come up by the end of the day with the correct result.

Those are great. So another question that people often have related to LLMs in general is how can they access data that's out? Either my data that I want to give it or, you know, data that's out there. And so Alan has a couple of things to say about how one can think about that in the context of custom GPTs. Might be having an audio issue here. Just checking whether or not it's on my end or whether we can quickly resolve.

I was just muted in more than one place.

Gotcha. Okay, yeah, let's take that one from the top.

Sorry, it's a pretty quick answer. Okay, great for people, for People who might be using Chat GPT already. And you know that in the input box, you know, you can click on the little paperclip here and you can, you can attach files, you can upload text files or whatever else. You can ask ChatGPT to, you know, look at those files or process them in some way. If you're using the OpenAI's own data analysis tool or just using plain ChatGPT4, you know, you can upload CSV files or other things there and it can, it can write some code to work with them. With your, Working with the Wolfram GPT, you can't do that. You can't upload a file here and have it get sent to the cloud. If you try that, we have tweaked the prompt a little bit, so it should remind you of this. But what you need to do is just have that data somewhere else accessible. If you could deploy it as a cloud object, you could have it saved in whatever your preferred cloud storage provider is, as long as it's accessible to somebody to just go directly to a URL and import it. Or certainly just accessing other data sets from the web, which is also something that I think that ChatGPT itself doesn't want to do. But you can, we've, we've opened up the import import function in the Wolfram cloud. If you're sending Wolfram language so you can paste in a URL of a data set. Just something I had done here. This is just, you know, a simple table of kind of vegetable price data from the usda. So just saying, you know, please help me analyze this. And it sends code to the cloud, it sends that URL, it imports that data set. You know, it's small enough in this case that it comes back and it's able to, you know, kind of see the data and comment about what are the columns, what kind of data does it have? You can then start working with it just through chatgpt. Show me the first few rows, it'll go get it again. Just run those out as a table so you can see exactly what the data is like. And then you could say things like, for whatever value it is, you know, give me a plot of retail price versus yield, for example, and just start doing some analysis. You know, one thing that I've, and then that's just sending more code to the cloud, again, working on that data set and generating a visualization. One thing that I have found really useful with ChatGPT is with the Wolfram GPT is if I have a Data set like this. Just asking the GPT itself for ideas about, you know, what, what are different kinds of analysis I could do on this data? And I find that it often does a really good job of, you know, coming up with a list of, you know, here are ways you can analyze this. You look at the distribution of values, you can group things by this column, you can combine these, you know, these are the different kinds of plots and visualizations you could do. So you kind of, you don't even necessarily have to of know what questions you want to answer. It's good at suggesting, suggest suggesting questions that you might want to ask of that data and then working through code to, to try to answer them. But really just the thing to remember is, you know that you, you can't send files directly from your system to the Wolfram GPT and have it analyze them unless they're, you know, very, very small things. If it's text files, the GPT can read them. If it's, you know, small amounts of data, it could send that data to the Wolfram cloud. Efficient thing to do is have, have your files somewhere else that you can just paste in a URL and import them through ChatGPT.

So we had an audience question that was about tweaking small things in the outputs, right? So if you ask it to show you code that it's writing, if it is writing code, or if you look at the, the input sorts of things, sometimes a hybrid approach is actually better than like, if you specifically, there's one thing that you want to do differently and you happen to know the language that it's writing in, and typically it'll just be some option setting in Wolfram language to tweak something like that, it's typically some kind of hybrid approach. Or if you ask it in natural language and you're specific about what option you want to change and what you want to change it to, that can also work. But sometimes depending on what I'm doing, I find that, that, you know, okay, it's done basically what I wanted and then maybe I just tweak one thing myself and then that is what I wanted. So depending on specifically what you're talking about, your mileage might vary for that.

You're talking about the quest. Somebody had asked about asking Wolfram GPT to modify the axis font earlier.

Yes, exactly right.

And I mean, you know, there are, there are some things that we're able to correct for, but there are lots of issues like that that are just, it's chatgpt making mistakes in Writing code. You know, one thing that I will say is, and again this, like, like you're saying it's useful to say, show me the code that you're writing. Look it over. You know, if there, if it's consistently failing with the writing code and it's using one function, you, you either know or suspect that it, it's, it's using a made up option or it's getting something wrong. There are a lot of those things that we actually catch in the API and we'll say, you know, this, this option is invalid. You need to do something else. This isn't a valid function. You can ask it to check the documentation for a specific Wolfram language function because we do have another utility API that is, it's basically using the, the Wolfram language symbol entity type and it can take any Wolfram language symbol and, and hit that API and it'll get back kind of a basic description of what the function does, some usage examples, you know what the valid syntax is, list of options. And sometimes that can be enough to get the LLM to say, oh, yep, I was making this up, this isn't the right syntax. I'm going to try writing the code again and it'll do better.

And we also have people who are asking about getting access to some of these notebooks. So typically after the streams we'll make sure that we've collected everything and they'll wind up on the permanent link to the YouTube videos that these go out on. So we'll make sure to organize that as usual. So if there's something that went by kind of quickly, don't worry about that. Another thing that we had talked a little bit about offline of going into was either thinking out loud or teasing a little bit. You know, how we might package some of these utility APIs that, that you've been mentioning that, you know, people I'm sure would be interested to write some of their own APIs. But also there are these APIs that you know, have been quite useful in making something like the, the Wolfram GPT work. So what, what should we say about that to people at the moment in.

The, like the Find Entity, Find Entity class? Yeah, yeah, I, I, I think we could, I could probably put those into a notebook and make those available to people too. I mean they're, they are, are as trivial as pieces of Wolfram language code as you could hope for. They just really go a long way towards letting the model do things more simply and efficiently.

Yeah, and again that's even though the large Language models in principle could be writing all this code themselves. You know, when you give them such a large menu of things that they could do, this greatly increases the chances that they'll do something not quite what you intended for them to do. So helping to give them very clear paths forward with some of these utility APIs can be a quite, quite useful thing to do. So also, if you're in the audience and still have questions, don't be shy about putting them into the chat. We will try to get to them here at the end here. And another couple of things that I'll briefly screen share just in case people have not run across them yet. So if you have not interacted with chat notebooks at all, which if you go to the Wolfram Cloud, so this is one place of course you can locally do this as well. But even if you have just a Wolfram Cloud account, if you click Chat enabled notebook you will get Notebooks. And Wolfram Cloud also now has wolfram language version 14 running on it. So it should have all the latest and greatest features. But along with that there's updates to documentation and things. So if you have not tried out chat enabled notebooks, then that use Wolfram Notebooks as their front end rather than a browser based thing. Although if you really do like browsers, there is Wolfram Cloud. This is a great resource. It's just introduction to chat notebooks wolfram.com language tutorial Chat notebooks and this does a great job of showing you some quick examples of things that you can do. And it will mention things like Personas and prompts which we've introduced some shorthand notation for being able to quickly without having to type out a prompt. Let's say that you want to very quickly. This is a nice short one. So let's say that we want to modify it to be some specific grade range, right? So maybe rather than typing out a long prompt that does this, you just want to use some shorthand notation. And so the prompt repository is perfect for things like that that could be be these sort of modifier prompts that can get put into the context that ultimately gets sent to the LLMs. Or it could be things that are more like defining Personas. So let's say that you want something which is particularly.

What would a good.

One to show real quickly be the design thinker I particularly like because it does a good job about scoping out a problem, right? So if you have something that you're just trying to brainstorm, the prompting that goes into this is nice for getting A clear path forward and then you can think about okay, so given this clear path forward, how can I create something that the LLM can use to actually execute this path and get me to some end result that I'm looking for? Another thing which you can look forward to more items coming up soon is LLMs are great at using tools and so if you write a tool that you think would be really awesome for LLMs to use, feel free to put it in our tool repository. There are some examples of some of these that are already in our tool repository that you can take a look at and look forward to more to come. So let me take a quick look again at our audience chat and see if we have any sorts of questions. So yeah, we have someone who's very happy about documentation updates. Yes. Yep, there's lots of new functionality in lots of areas and quite a few functions have gotten some significant interesting updates. So do check out the new documentation that should be live now. So we have another person who's interested in doing a live living dangerously. Let's see, what kind of example do you want to do? We want to take one of our examples that we have from before and run them again and see what sorts of interesting things happen.

I don't know if Michael isn't up against the limit for GPT4, but I think I may have hit my usage cap for the next half hour or so. I think I was trying to make sure I had everything in before we.

Started the live activity to do chat GDP live. Yes, it's. No.

Well, so while we're taking a look at. Go ahead Michael.

Yeah, no, go ahead. I was going to say I find it Quite interesting what ChatGPT can do right now. Already writing Wolfram language code. I'm really sometimes quite impressed what it can do. So. Okay, so here was a nice archive preprint hitting a prime takes I think it's a cool idea like you roll the dice over and over and how long do you have to roll if you stop when you have have twos you will roll forever but chances will not and really hard problem but you can say this and you ask it to analyze the problem it writes a page of code and I told a little bit what I want. I say I want something that basically makes an infinite loop in a dynamic and it does do a great job really building the things and let's evaluate four reasons that I don't know. Yeah it's really 2.43 and one day.

Looks like we had another brief. Another brief Audio hiccup on, on our end, Michael, if you could maybe you were about to say one other thing and then it cut out. So if you could perhaps pick up from there for our audience. Well, while we're, we did have another question while we're getting that audio issue sorted out. This is a really good question. So what other sorts of custom GPTs do you encourage users to develop using Wolfram language APIs? While we're waiting on the issue with Michael's audio to resolve, Alan, do you have any thoughts about that?

Take your pick. I mean, I'd be really interested to see what, what anybody else develops on this front. You know, I think we've, we had already seen with plugins and I can see starting to browse through the sort of embryonic GPT store that there are some other people out there who have started to write their own custom GPTs that are hitting the Wolfram Alpha API in the background for the most part. I can see a lot of things that are doing that right now. I, I don't think I've seen anybody who's, I haven't seen any examples, but I haven't had time to look at for a lot of examples of other people who are leaning on Wolfram Cloud APIs as the basis of a GPT. But I mean, I think that's, that's the easiest place to start. And there's certainly a lot of areas where, you know, the, the built in code analysis, data analysis functionality and ChatGPT, you know, has some, has some gaps that I think we fill nicely. I mean, certainly a lot of the, the GEO functions, there are things that are really powerful and hard to duplicate even with other kind of Python packages that are out there. The other thing that I kind of notice just, just more anecdotally than anything is Wolfram code. The, the amount of code that's required to do something, whatever that something is. Often the Wolfram code is a lot more compact than a comparable chunk of code that has to be written by, you know, and not to, not to, not to, you know, slag off the code writing capabilities and the code analysis interpretation stuff of ChatGPT itself. But when you're worried about conserving tokens and you've got to kind of keep things as tight as possible going in both directions. You know, I, I've just had a really good experience with the speed with which code can be written and evaluated and things being sent back and forth. And as Michael says, you know, to get a good result with anything with an LLM, you need to talk to it for a while. And so when you're. You're adding that up over multiple steps, I see a lot of benefits there.

Yeah. And as a reminder to people, make sure that as you are thinking about what are some interesting APIs that you can write, definitely put that function in the function repository too, because that's a great way of expanding the capability that everyone has as a community. And another advantage about writing concise code is that then if it's not behaving quite in the way that you expected it to, if it's concise, it's quite easy for you to look at it as a person and say, oh, I suspect there's a mistake happening here. Whereas if it's a very long piece of some elaborate thing with syntax that has tons and tons of syntactically meaningful things, but it's not necessarily compact in a human expressive way, then that can be difficult to unpack what exactly might have gone wrong. So it looks like we do have Michael back from the audio issue, let's hope. Yes.

Sorry about this. I have no idea what happened.

No problem at all. These things happen. So it cut off for you right around the time when you were about to say it just gotten to the result where you'd showed that in fact. Yes, you know, you get the results of 2.43, I believe it was. And then you were about to say there was one other interesting thing and then it cut out. So I didn't get to hear after that what the other interesting thing was.

Okay, so then let me share a second attempt. But he has to unshare first.

My mistake.

With respect to code writing. So that was just today. There was a post a devolved From Community just 7 hours ago from Sundor who always makes very nice can chatgpt make this? And I tried it this morning and it's actually quite nice. So dotikahedron then make outwards oriented vertices cones and I want to change the length of the cones, the color of the. And it made this. So let's go there. And it wrote this piece of code that I literally took, did nothing, no variation, and it works just fine. I find it quite remarkable that an LLM can do this today.

That is amazing. There's a couple other interesting audience questions that are quite good questions. So one of them has to do with, you know, the fact that Wolfram language is meant to be something that humans can express their computational thoughts in. Right. So I think the, the biggest benefit there actually is on the side of human interpretability. Right. So you can fine tune large language models to write in all sorts of languages. Right. But I think then the question is what's the most useful, given that you want to fine tune it to write in some language and then the human might be interested to actually read the code. Alan, Michael, do you have any thoughts about that point in question?

Sorry, could you repeat the question I missed?

Yeah. The question was, so what? How does one think about the fact that Wolfram language is meant to have a very expressive character in the context of how does this work in the LLM world? In other words, if you read Wolfram language code, it reads oftentimes a lot more like a natural language sentence than many other languages. So the question was, how does one think about this in the context of LLMs?

It's interesting. I mean, I don't know if we know yet how much that's a benefit. I mean, I certainly see it. Right, good compact code. I guess the, the, the one downside is that I, I see that Chat GPT because it can kind of read more like natural language, it feels very free to make up its own functions that look like totally plausible Wolfram language functions. Although that, that has also been something we've, we've kind of made use of that over the last year. Right, Michael?

Yeah.

Sorry, go ahead.

No, no, it's a very good question. I'm not sure kind of if it's a pro or not. I think the future will show.

There's another very good question speaking of the future, which is how far away are we from having LLMs be able to read multiple documents, research documents, and synthesizing these ideas? So actually already it can be quite useful for summarizing these sorts of things. Right. In fact, one of the tools that will be showing up soon, I hope, in the LLM tool repository and happy to show anybody. Unfortunately I don't have it on my machine at the moment, but so you can take for example, archive papers and have it synthesize what are a summary of this and you can have a nice user interface for clicking around. So there's no question it can be helpful for taking ideas and summarizing them and presenting them in ways that are quite helpful and perhaps faster than if you were trying to just dig through these documents yourself. Now the question is, can they synthesize things from across multiple domains? That's a much more subtle and I think time will tell kind of question. So, Michael and Alan, if you wanted to opine on that particular point, it's a very good question.

Not particularly Now I think we're getting.

We'Re getting a little thinking.

Yeah, I think we're getting into just sort of what's next for LLMs. I think maybe we're veering a little bit away from the basic stuff we're with today. I think we're all interested to see what's coming, but Michael and I are certainly not the ones that are driving LLMs forward at that level. So.

So any other audience questions? I think we've, we've reached pretty much all the audience questions that I'm seeing here in the chat. If anything else comes up here next in the next little bit here. So as a reminder, do check back to the permanent link to this video and you should have links to many of the things that we have shown here. So, so some highlights to make sure that you go and check out. Let me so there is the Wolfram GPT, which if you're a OpenAI plus user, you can have access to by following these various links. The introduction to chat notebooks, if this is functionality that you're interested in trying out, has a great tutorial here. Don't forget things like the prompt repository. And if you're someone who is writing APIs as endpoints for these custom GPTs and has some interesting nice functions that you've written, do feel free to put them in the function repository because that can really help the community. So I think, let me just do one last check to see if we have any new audience questions. We have some thank yous. So Michael, Alan, thank you so much for joining us and we appreciate all the audience's time and your time and thanks so much for tuning in everyone.

Fun.