ChatGPT Gets Its Wolfram Superpowers

Sa.

Hello everyone. Just doing a quick mic check here. The Daily Study group is going to get started in about four minutes at the top of the hour. It's our final session and we're going to be covering that last part of the book on Chat GPT gets its Wolfram superpowers. I'll get that pop up offer for the book link in to your screen here shortly. Get questions in that Q and A pane and we'll start here in a few minutes. Thank you.

Sa. Sam. Sa. Sam.

Hello everyone. Welcome back and thank you for joining us. I'm Cassidy with the Wolfram youm Team and today is our fifth and final session of the Daily Study Group. What is Chat GPT doing and why does it work? We actually have Alan Joyce today to go through many examples from this last section of Steven's book and Arbin here as well. We'll touch on some final items before we close out. I did want to go ahead and make sure that everyone knows that the quiz link is available there. It's in your recording email and you can take and pass that quiz. Oops, sorry, one second here. You can take and pass that quiz at any time before the 20th of September and we will be able to get you a completion certificate for the series. I've also put a link to the. Excuse me, to our Daily Study Group page. Overall that's one to just bookmark and go back and check. We're always adding new ones. Right now I think we have. We have Monday starting a three week elementary Introduction to Wolfram Language and then following shortly after that is an Introduction to Complex Analysis for late November and then we'll probably start back up in January. Also a link to the Machine Learning or the Exploring AI Foundations with Wolfram Tools Core sequence. Actually this week we did the Introduction to Machine Learning lesson but you can always go and re sign up for that. But we've got one on Neural networks and one on LLM tools that we're going to be showing. So we hope that you sign up if you want to kind of continue this AI and LLM discussion. But there is tons to discuss today. So I think I'm going to pass to Arvin first, I think to welcome Alan and say what he's going to be covering. And yeah, I'll be back at the end.

Arben.

Yeah, all right, just briefly. So as I mentioned yesterday, Alan is from the Wolfram Alpha team and he'll be telling us about some of the integrations that we've been kind of hinting at these Last couple days between Wolfram language, WolframAlpha and ChatGPT. So, as you might recall, there are a couple of fundamental issues with LLMs. The two primary things are largely that they don't have a way to really understand what it is about the world that's actually true. They just kind of know what's in their training data. And then the other problem that they have that's kind of a serious one if you need to use them for serious work, is that they're not so good at doing directly algorithmic tasks. And it just so happens that a computational language like Wolfram language addresses both of these because it has factual information about the world which is human curated, not kind of randomly quote, unquote generated or even web scraped, but actually curated. And then of course, fundamentally Wolfram language is a powerful tool for doing computations. That was its original intent after all. So by adding these two pieces of functionality to ChatGPT, you can really extend its functionality. That's kind of a tautological sentence, but it's true. I'll pass it to Alan to walk us through some of this integration.

Great, thanks very much, Arban. Hopefully you've got my screen here. So my talk after Arban's excellent coverage of Stephen's book, the rest of this week. This last part is relatively brief and simple compared to this plumbing of the depths of what's going on really on under the hood with ChatGPT. This is also the part that really has changed quite a lot since the book came out, since that first blog post came out describing ways of giving ChatGPT its Wolfram superpowers. When this first came out, we were in the kind of pre tool calling, pre LLM agent kind of era where what we were trying to demonstrate then was, hey, if there's a way of directly linking these things together, Wolfram Alpha Wolfram language could, as Arvin said, help to compensate for some of the inherent shortcomings of LLMs. So if we kind of go back to the beginning here, when we were seeing the first ChatGPT models come out, everybody was kind of amazed. This is the first time you're seeing this really convincing human, natural sounding text, write me in it, say about anything, and you get something that sounds very much like, like a person would write, you know, not very robotic. But for us, you know, we're, we're dealing with, you know, computational, you know, real world data, real world knowledge about things. It was easy to see where the cracks were. So, you know, just asking a question of ChatGPT, you know, asking a direct question to kind of see what its grasp of the facts was. You'd find, you know, the distance from Chicago to Tokyo. Here's this very, you know, nicely worded, verbose, sensible description of things. But fundamentally this is wrong. This is off by, you know, a thousand or so miles. So we were thinking at the time, you know, this is, this is exactly the kind of thing that WolframAlpha excels at. You know, we give it a natural language input, use our parser to turn that into a semantic representation of that input, turn that into a query that, you know, looks up some data, does some computation and gives a very precise result. So, you know, the difference between those two things. This, this sounds very nice, but it's just fundamentally wrong. Here we get the right answer. It's something that's reproducible. You can understand how the computation was done. And so when the book came out, if you're, if you're reading the book, you're going to see a lot of kind of very hacky early ways that we were thinking about bringing Wolfram Alpha and the Wolfram knowledge base into ChatGPT. So this is one example that Stephen had in the book, you know, tell me about livestock populations in Turkey. And what this did was, you know, here's a nice overview. It's a major producer of these things. It's claiming to give you data from the FAO as of a specific date. You know, this seems very authoritative. It's, it's sourced, it's got real numbers. But then we were kind of realizing this is, this is not actually the correct numbers. You know, if we looked at the data from the FAO, which we had in Wolfram Alpha from 2020, these are not at all the correct values. So what we were suggesting was this kind of workflow, you know, ask cheap chat GPT to come up with basically Wolfram Alpha queries. So here are all the facts in this text. You know, how would you ask Wolfram Alpha to verify them? Number of chickens, number of turkeys, blah, blah, blah, blah, blah. Suggesting, you know, we can feed those into WolframAlpha here. Just using the WolframAlpha function in the Wolfram language gives you kind of a more direct programmatic interface to WolframAlpha. So feeding in all those inputs, asking for a plain textual response. Because this was something that we wanted to kind of have this dialogue with ChatGPT. And we're thinking ChatGPT, it's, its inputs and outputs are human text, you know, human speech. So let's have WolframAlpha produce data that we can feed back to it in that kind of way that it can ingest the data most easily. And then we were saying, you know, you can insert this into the original response, highlight them, and then, you know, this is great. Here's ChatGPT. It's the original text that it came up with, but we've given it actual factual data from our knowledge base, and it's able to insert that in here. And so you get kind of the combination of the factual content from our knowledge base combined with this very nice, you know, readable human way of tying it all together and presenting the content. So since then, you know, this. This changed very quickly, 2023, I think, when this first came out. And so what we were seeing was, you know, then we start seeing plugins for ChatGPT. We start seeing people figuring out how to let LLMs make tool calls, so calling external APIs, doing other kinds of computation, pulling other data in from other resources. And so, you know, very quickly, with the original plugin for ChatGPT, and then the custom GPT, this is something that just happens automatically under the hood. You use our custom GPT, you say, tell me about livestock populations in Turkey. ChatGPT says, this is the kind of thing that I can get this information from WolframAlpha. It makes a call to Wolfram Alpha, it gets the data in, and then it can give you a response. You know, here it's being very terse, but you could say, you know, give me a little bit more of a verbose response and get something there. And so this is great that the result here is now you've got access to a huge amount of reliable curated data and. One second, I think I need to. There we go. Just wanted to share some more examples here. Sorry, is my screen not changing?

No, it's static. On the. On the notebook.

Yeah, one second. Here we go. Okay, so with the Wolfram GPT, then, you know, what you've got is access to all of this content. So the GPT, ChatGPT, with that custom version, knows I can call WolframAlpha. I can send natural language queries and I can get, you know, reports back, specific results. I can get, you know, precise date and time conversions. I can explore anatomical data. Asking questions like this, which ChatGPT could probably give a good answer for this because it's more kind of conceptual. But, you know, Wolfram Alpha, you get the benefit of other kinds of computations and being able generate visualizations on the fly for things that you're Interested in and certainly, you know, mathematical functions, doing any kind of math and being able to get plots, charts, maps, all these kinds of things. So having this kind of direct, you know, plug into, Second. Into everything that Wolfram Language and Wolfram Alpha have to offer was a huge step forward. Now, in order to make that happen, what we had started to have to do was create new APIs that were designed to be called not by people. We've had APIs for WolframAlpha that returns essentially the full results that you get on the web. We've had APIs that return just spoken results or short answers, or all kinds of different ways of getting Wolfram Alpha results back. And so we had started with thinking, well, what does. Again, an LLM is generally a big machine for ingesting text. Certainly now we've gone into multimodal models, but at the time we were focused much more on text input. And so we came up with a WolframAlpha API for LLMs. Not that there's an LLM under the hood, but basically this is an API that we meant for an LLM to be able to call and get data out, that is in a format that's going to be most convenient and efficient for it to ingest that data and then work it into its results. So for an example, this is WolframAlpha, a web result just for an input like Solar Eclipse. And you get, you know, there's an input interpretation, there's a result, there's all the other kinds of entrained data that you get there. We generate some visualizations. The API for LLMs, this is the result for the, for the LLM. So it's basically kind of a plain text but still semi structured version of the WolframAlpha output, including things like assumptions and with kind of tips for the LLM. You know, if you wanted to use this a different way, like maybe you're just looking for the definition of Solar Eclipse. This is what you need to pass back into the API. But then, you know, this is the precise interpretation of what we think you were looking for. And then here's all the data, you know, down to if visualizations were generated, then here's a URL where you can go and you can grab that visualization and then, you know, ChatGPT, the web interface. Other LLMs generally online, you know, are very good at working with generating output and markdown and so they can just pull in a URL, show that in Markdown it gets included in the results that's showing online. The Wolfram language API for LLMs, a little bit different because this was an API where we thought, okay, we're seeing LLMs, you know, especially at the time ChatGPT. Certainly other LLMs have also been getting much, much better at writing code of all kinds, including Wolfram language. So what we wanted was an API that would not just let an LLM send Wolfram language code and evaluate it and get a result back, same kind of way. We want an API that's going to return not just the results, but also kind of helpful information for the LLM. Knowing that often when you're in kind of a situation where you're doing looking for code assistance with an LLM, the LLM can need help. You know, it doesn't necessarily get things right. And if you, if you know a language really well, then you've got kind of an advantage because you can say, well, I think you did this wrong. Try this. If you're not really familiar with it, then it can be a lot harder. So we have things like the Wolfram language API for LLMs. It'll evaluate code, but there's also some other things happening on the back end. There are certain common coding errors that we were noticing that LLMs would make. There's a little bit of autocorrection, sometimes just simple things like matching brackets around code or other things where they're just sort of common errors that LLMs would mix idioms from one language with those in another. Because it's been trained on all this code in all these languages and sometimes kind of thinks it can mix and match more than it can. So, for example, this is one query that would be sent to the Wolfram language API for LLMs. And this is pretty straightforward. It's valid Wolfram language code. What comes back is, here's the output. It's a link to an image that you can display and markdown. Here's code that has an invalid option. And so this is something where we don't just saying that's not a valid option for this function. We also have sort of recommended solutions. So to make sure that LLMs don't kind of get caught in a loop where they just keep saying. And if you've done any kind of interactive coding with an LLM, I'm sure you've seen this, you know, well, that didn't work. Let me try this, let me try this, let me try this. We wanted to steer it in a better direction. So this would be saying, you know, this isn't a valid option. Maybe you don't know what the Options are, here's a suggestion, you know, send this code and you're going to get all the valid options for that function or things like this. If you're working with entity value data, data that's in the knowledge base that's integrated into the Wolfram language, then, you know, here's a case where this is an area where we see a lot of hallucinations coming from LLMs, where you say, you know, write some Wolfram language code to get, you know, the height of the Empire State Building and it'll make up some canonical name for the building. So here's a case where we're saying, you know, here's, this is a code, this is not a valid entity. If you want to find a valid entity, then, you know, submit you know, the name of that entity to interpreter, and you're going to get something else. We had some other things that we, we developed here also that switch one more time here, Where we developed some other kind of utility APIs. So for things like this, helping the LLM know how to get correct semantic representations of kind of more complex kinds of data lookups. So this was an example like this is using some of these utility APIs, use Wolfram language to find the five tallest buildings in New York City, compare them in a bar chart. ChatGPT goes, makes a few calls to the cloud, and then comes up with a bar chart. It doesn't show up because of how OpenAI handles sharing. It doesn't have images, but here's the data. And so then if you ask it to explain this, what it was actually doing is saying, okay, I need to find the five tallest buildings in New York City. Rather than trying to, you know, right from scratch, a complex implicit entity class expression, we have some prompting that says, you know, just send a natural, simple natural language input to a semantic interpretation API. And what it gets out is the semantic representation of that input. So then it can take that input, feed that in, get the properties out that it wants, you know, we want to get the height, we want to get cities, and it gets the data. And then from there it knows, you know, it does a good job of just writing a bar chart that's going to take those values and it ends up with an image of the bar chart. So, you know, this kind of progression here that we went through, you know, figuring out what are, what are the things that someone wants to be able to do with an LLM, what are the ways we can make it easier for LLMs to basically put their hands on Wolfram Alpha and Wolfram language and really streamline that process. So we came up with all these other custom APIs for things. And now of course, that has gone a step further. And really the best place if you want to see ChatGPT or any other LLM enhanced with Wolfram Superpowers is in chat notebooks that Steven has shown off. We certainly have other courses and resources online to go through these in detail. But now we can go back to that initial query from the original blog post here and say, here I'm in a chat book setting. This gives me access to a few different Personas and we can get into those a little bit more later. You can add other ones. You can add tools that the model can call here we're just using an OpenAI ChatGPT model, but you could certainly go in here and if you have an API key for LLMs from other services, there are a lot of other things you can get direct access to. But Anyway, so asking ChatGPT in this case, how far is it from Chicago to Tokyo? Same thing's going on under the hood, calling the ChatGPT API, getting a result back here, which today is a whole lot closer to reality. I mean, this for things like this sort of basic facts and things about the world, especially when you're dealing with very well known places like Chicago and Tokyo. Certainly somewhere in the massive amount of training data that ChatGPT has ingested, you know, this, this kind of information is somewhere in there or something that allows it to give a reasonable approximation. Although still that requires you to put a lot of trust in ChatGPT. And so, you know, we could follow on from that input and say, well, how did you come up with that answer? And so it's saying, you know, this is, you could use the concept of a great circle distance and calculate it this way, but actually what I did was just use commonly available data. So again, this kind of gets at the heart of LLMs can be really good at writing text. They definitely have in their training data, lots of valid data about the world. The problem is just sometimes things can be a little bit off, sometimes things can be a huge amount off. And if you're doing any kind of, you're engaged in any kind of pursuit where you really care about precision and accuracy and verifiable and repeatably verifiable facts, then what you really want to be doing is using Wolfram tools so that you can do that kind of stuff more automatically. And so in contrast, same kind of query in this case, this is using the code Assistant Persona up above, this was using just the RAW model, really nothing else coming in between your query and the API call here. This is a different Persona that has access to a number of other tools. So you know, the RAW Persona has none of these attached. The Code Assistant has some additional tools it can call. So there's a documentation lookup, the documentation search. It can make Wolfram Alpha calls, it can evaluate Wolfram language code. A few other additional things that I can, I can talk about later. But as a result, you ask, how far is it from Chicago to Tokyo? It knows, okay, this is a computation. This is something that I can use geodistance and I can get a precise result. It writes the code. It even incorporates, essentially control equals inputs, natural language inputs for Chicago and Tokyo. So it's not trying to write out the canonical names for those. It's using the WolframAlpha parser to turn these places into their semantic representations. And it's evaluating the code. And you can pop this, open this Wolfram box, you can see the interpreted inputs, you can see the results, and then you get a very specific, reliable, verifiable computed result. And then if you want to understand that better, you can even delve a little further and say, what's actually going on here? And because ChatGPT in this setting has more access to Wolfram documentation, it can give you a much more detailed explanation of, okay, this is what that function is actually doing. You want to see how that function works. You want to understand some of the details about what's really happening under the hood. Here you go. So this is obviously really powerful. Now, you know, going back to Arbind's comment from the beginning, you know, certainly you're, for the most part, people are not firing up ChatGPT or any other LLM because they want to use it like a calculator. But it is worth kind of paying attention to how good LLMs are at being calculators to understand just how far from the truth things can get. When you're doing anything more complicated, you know, basic math, what's that? You know, it's a fairly large exponent, but what's 3 to the power of 73? Calculating that results in a very large number. No surprise. Using a calculator or a computational tool, which it hasn't actually done, you get this number, which is absolutely incorrect. In contrast, you know, use the code assistant. What's 3 to the power of 73? Very simple. Let's evaluate it. You get an answer. This is clearly not the right answer, or certainly if you want to gather other real world information that's relying on, there's some temporal aspect, there's some geographic aspects, there's other kinds of computation that are involved. Ask an LLM what constellations will be visible from Chicago at 10:00pm tonight. And this is not a bad answer. This is kind of hand wavy. You have to consider this. Here are some autumn constellations. This may or may not be accurate. In most cases it's not a bad result. It's a typical kind of LLM generated web summary kind of result. Probably not what you want if you really want a very specific answer. And so you know what constellations would be visible from Chicago. Here's a case where this turns to the WolframAlpha tool and it'll just make a query to WolframAlpha. It gets the results. Here's a long list of things telling what's above, below the horizon, rise and set times, a sky map. And so you get something here that's much more precise, accurate, detailed, you can understand where the data is coming from. So really useful kind of thing. And then certainly when you're getting into, you know, trying to write code, and in some cases it may not, it's not really that you, you want to write code, it's that you've got a particular problem you want to solve, you've got something you want to be able to visualize, you've got something you want to analyze, but you're not really sure how to go about doing that. This is the thing for me where this has been the most powerful and the most useful on a daily basis, even as someone who's at least mediocre level in writing Wolfram language, to be able to have this tool at my fingertips to come up with things very quickly. So a basic question, and this could be sort of more of a factual thing. I'm just trying to understand this concept in general. What's the Eurodollar exchange rate been over the last year? And so another case, this is one where here it turned to Wolfram language and it's pretty straightforward. Dates, making a financial data call, making a date list plot. Here's a value. And you know this, this is kind of showing what's happening under the hood, pulls it out, shows you a nice result in your chat session. And so then I'm thinking, well that's interesting, but maybe I want to be able to work with that data a little bit more. And so you can build on that and say, can you help me turn that into an interactive object? So it Does a pretty good job of this. It writes some code, turns it into a manipulate, explains what it's doing in all these parts, politely asks me if it wants me to evaluate it. So it does, and I get this out and it's pretty good. It makes sense. Although this is kind of strange. I can see how it's sort of the, the date ranges I was asking for, but it's, it's chosen to do time ranges as fractions for some reason. And so this is another thing, you know, build on what you've got. That's not exactly what I wanted. Those are kind of weird. Why don't you use real quantities also? Because I just happen to know that this is not necessarily, and is another one of those kind of odd artifacts that has come out of its massive training set. I'm just telling it, you can drop that. So it goes back in, keeps everything that was good changes the things that were bad, and you get another version of this out. That is a nice functional manipulate showing, you know, currency exchange rates over the last six months, one year, five years for a range of things. So this has been one of the things that's, that's really kind of powerful for somebody who, you know, you might have a specific coding project or you might just have a subject that you want to explore. And having something at hand that can say, you know, there's a way we can visualize that, there's a way we can analyze that data. I can, I can write a tool for you or help you create a tool that's going to help you understand this topic better has been the really powerful thing. And certainly these can be much more elaborate. This was an example that our colleague Michael Trott had come up with kind of early in the days of looking at ChatGPT and playing around with a lot of different kinds of Fermi estimations. Very complicated problem. We don't really think we can get a precise answer. But let's think through how you would estimate these values. And this is kind of interesting. It's the kind of thing that an LLM can do pretty well on its own. But that's still another case where it may come up with numbers and do calculations in the course of this very long exploration. And, you know, very small errors can get compounded over time. And so this is the kind of thing that's been really, I think, interesting and useful for me, having the Wolfram integration where, you know, here, help me through figuring out how to estimate how many photons from sunlight hit the back of the eye of every person in Pittsburgh in an average day. Kind of crazy, but it walks through. Okay, these are the steps. I want to do this. I want to do this calculation. I'm going to make some assumptions about certain values. You know, all these kinds of things that it's going to kind of walk through. So doesn't always get what it wants. Here was a case where there's a value we didn't have in WolframAlpha. So it falls back and it has an assumption that's pretty good one. It makes a reasonable estimation of a value we're going to use there. That makes sense. It has certainly some valid basic formulas that it's already gotten its training. These are, ok, let's make some other assumptions. And then this is a really useful thing where I've seen a lot of cases of this where you've got these long reasoning chains and it really only takes one or two places where things haven't been calculated quite right and the final result is just really far away. It's very hard when you don't have some tool integration here where you can go back and you can look at those steps, you can look at the computations, you can be sure that the value that you got in this step was actually carried over into the next one and it was a correct value. And so all these end up getting evaluated. You know, here's all these different calculations. You know, I get this value, I'm going to feed this value into the next stage, I'm going to feed that value into the next part, and then you end up with kind of a reasonable result at the end. So these are always kind of interesting. And then beyond that, you know, certainly there are a number of other tools that we've developed along the way. And so there are things like the prompt repository. And so some of these are just kind of fun. Some are very useful. If you're working with chat notebooks, you know, there are things here that are. A lot of this is focused on kind of text manipulation. But let me see, there were some. Some ones that I always enjoyed here, like, yeah, legal de jargonize. Just some helpful little things where you have prompts where you might have some task where you are repeating some sort of operation. And this is useful because there's some, you know, good kind of clear prompting for a specific task under the hood here where you've got like in this case, big chunks of legal jargon that you want to represent in kind of natural language. Exactly the kind of thing that LLMs tend to be good at but it can be hard to get a prompt that really does a good job of giving you exactly what you want on the output. There's also a small, and this is still kind of in development, but what I was pointing to before, there is a tool repository and you can click a button here and get a notebook that you can use to build your own tools. And this has been a really powerful thing I think for anybody who's using LLMs locally for specific tasks. This was one that I had a use for actually in kind of doing some, some Wolf Malpha work was we realized kind of early on that ChatGPT, other LLMs, even though, you know, they've been trained on, you know, basically all the, the open content on the web, including things like Wikipedia and Wikidata, that they still will hallucinate facts from those things. And so we were kind of trying, you know, can you, can you do a web search or can you write a sparkle query to get data out of Wikidata for some kind of complex query? And ChatGPT would hallucinate if you know anything about Wikidata, you know, sort of items and properties are represented with these numeric codes. If you want to look things up, you need to know them. And ChatGPT would just confidently look up totally invalid or wrong codes. So this was kind of a funky tool that I had written at one point that taught it how to use Wolfram language functions like Wikidata search and Wikidata data to look up the correct property and item identifiers and then run some of those queries to get data. So you can do things here like you plug in this tool and you can say something like I want to find information on left handed female politicians in Wikidata, which is something that we couldn't right now answer say in Wolfram Alpha. And so this is actually a good example. It actually thinks that it has come up with the right properties on its own. Sorry, I think I just realized I hadn't switched my screen here, but let me go over here. Sorry. This is a LLM tool notebook. And so asking the LLM this question and then it actually hallucinated at least one property in here, didn't work. So then it did what I told it to do, did a search for all those properties, came up with the right identifiers and then ran this query and actually got the data I was looking for out of wikidata. So this is a very kind of wonky, very special purpose tool that I had Some use for, but it's the kind of thing that's really, really powerful here is that it's not just ChatGPT or any other LLM + WolframAlpha and + Wolfram language. It's those LLMs + Wolfram Alpha, Wolfram language and all of the other things that you can build with Wolfram language to extend the built functionality. So resources from the Wolfram function repository, data from the Wolfram data repository, these additional prompt resources, LLM tools, all this stuff that you can dig in and, and kind of build on what's already there. And so there's certainly other LLM related functionality. I think that I'm gonna, I'm gonna kind of wrap it up here. But anybody who's doing any kind of work, it's the chat notebooks are sort of the easiest interface on here where you're really just having a conversation with the LLM. But there's a whole lot of other tools available for people who want to write code, you know, repeatable tasks that you want to run that leverage some of the LLM power under the hood somehow. So writing functions that are built into other kinds of arbitrary code that can send data out or ask an LLM to perform some operation. Newly released tools In, I think 14.1 with doing more with semantic search and building semantic search indices and using those to extend the knowledge of LLMs in particular areas. So a lot of functionality here that we've built on and you know, I, for me, this has been by far the best environment to be exploring LLM capabilities in and building on what they're all already able to do and combining them with what Wolfram Alpha and Wolfram language bring to the table. So I don't think that I've got more that I necessarily need to cover. I don't know if there are questions. I haven't been tracking the questions on the, on the stream very closely. I don't know if there's anything that's still open there that should. People would want to cover or arbin if there's anything you want to come back to here.

Yeah, I think what we can do is if people have questions for Alan about any of this functionality, they can go ahead and ask them in the Q and A pane. Now, I will probably point out a couple of things that we have addressed at least shortly in the Q and A pane, but I also do have a question for you to get this started. So when you use these integrations and you say something like, oh, okay, it knows here that this is a computable question. And so it's going to make a call to Wolfram. What is it that keys off the LLM that. Oh, this is a question that Wolfram will be good at answering.

Generally. Prompting that's under the hood. You know, if we're talking about the custom, the original plugin for ChatGPT, the custom Wolfram GPT or when you're dealing with chatbooks, there is additional prompting that's gone into each of those things that gets invoked when you start a chat session. And it tells the LLM, you know, these are the tools you have available. These are the kinds of things that you can use these tools to accomplish. I mean, it helps sort of in general. You know, any large LLM kind of comes with an understanding of Arpin or Alan.

If you guys share a screen, there's like nothing being shown right now.

Sure. I don't really just even, just even.

A chat GPT scream to be fine.

I'll go, I'll go back to the, back to the, the notebook. Sorry, I think that, Sorry. The, any large LLM I think kind of knows what WolframAlpha is, right? There's. Even if it can't, you know, access the Data that's in WolframAlpha, there's an understanding just from all the training material that's gone in. WolframAlpha is a tool that can take natural language queries and it can answer questions about math, geography, music, sports, whatever else. You know, there's, there's an understanding, you know, the Wolfram language is, you know, they certainly have an understanding of what functions are in language. They can write code, they understand the scope of kind of the integrated data. But there's definitely prompting in all those things to kind of guide the LLM through how to think about writing code, how to make a decision between. Should I call WolframAlpha? You know, is this something that I can, I just want a simple answer for, like, do I, do I really need to do a complex computation here? Or is this something where what I want is a quick answer and some other context from WolframAlpha? Now that making that decision is kind of the first thing in a lot of the prompting and the rest is, you know, more fine tuning and some suggestions about consulting, documentation and particular kinds of errors that LLMs tend to, to make.

Okay, that makes some sense. So, you know, there's some built in information just from the training data about, you know, what's possible with these languages. And then there's extra prompting about, about this and the prompting is enough because, you know, with the reasonably good understanding of language, if we can call it that, LLM has, it can kind of make the connection between. Oh, okay, you're saying that for these types of computations I should do this. And so when I look at the, when I look at a particular prompt, I can kind of classify that into one type or another and determine whether I should be using a tool for it.

Yeah.

Okay, that's, that's pretty neat then.

Yeah.

Okay, let's see if we have a couple questions from the audience. So there's a question from Lucas, and this is something that comes up pretty frequently, actually. Maybe you can offer some clarification on it.

Sure.

So the question is, I'll just share it on screen. What is the recommended way to make this chat? Interaction between, I assume, Wolfram functions and ChatGPT. So should you use ChatGPT with the Wolfram GPT plugin? Should you use a chat notebook or is there some other way? Or why would you pick one versus the other?

I think it's what you have available. Certainly if you've got Wolfram language access, that's always going to be my, my first preference. It's easier to switch between models. It's easier to having access to a chat session in a notebook. And maybe this is something that's worth kind of showing here. One second. I think the. Sorry. Okay. The thing about, you know, dealing with like the Wolfram GPT, if you're going on the web, you know, you can, you can start a new chat, you can have a chat, you can kind of copy paste things out of that chat and work on things elsewhere. But for me, just having the notebook interface is just enormously helpful for things like I should just go back down here. The code assistants here, this kind of interaction where I've got a question, it's writing some code. This is the kind of thing you can do online. You certainly can't see interactive plots on the web. You know, the, the API and the ChatGPT website can only show you static things. Right now, the fact that you can, in one notebook, you can also do things like, you know, I can insert a delimiter here. And so I've got this chat up above and that's kind of one chat session. But I could start another one down here and I could, you know, I could have this other session in the notebook, use an entirely different model. I could have a session down here, use a different set of tools, and it's going to be sort of isolated from this thing up here. So for it, I think that the best, you know, the best setting for me is certainly working in a Wolf notebook, and that could be desktop, it could be Wolfram cloud. Certainly you can access ChatGPT things there. You do need, you do need to have a OpenAI API key to be able to use things in notebooks. So sometimes people, I think, still get a little bit confused by, you know, I have a. I have a ChatGPT plus subscription. How do I plug that into Wolfram? That's not really how it works because the chat notebooks are using the API, which is a separate thing from the ChatGPT web access. But certainly the notebook environment is just. It's more powerful. I think the. For me, it's a better choice. But there's, there's cases where the web is fine.

I think, I think that I kind of align with that as well. You know, of course we have access to Wolfram language, so it might be easy for us to say, but certainly if you have access to Notebook is a nice environment. One more thing about working in chat notebooks that I don't see mentioned too often is that in a chat notebook, you can use all these different models and everything, right? But you can also just type straight up Wolfram language code yourself. And the thing about a chat notebook is that usually in a Wolfram notebook, the kind of order that things appear in the notebook doesn't really matter. What matters is the order that you evaluate your cells in. But in a chat notebook, whenever you do a chat input, it looks up. In the notebook, wherever your chat input is, it looks up all the way to the beginning or to one of these chat delimiter cells that you see in Alan's notebook there, the little gray bar. If you've written a bunch of code and you've said, I've saved this image as my cool image, you can then in a chat cell, say, could you help me to binarize my cool image? It will have access to my cool image. It will actually embed the code that makes up my cool image in the embeddings that we've talked about throughout the week and use that as an input. What it can do is that it can say, okay, yeah, sure, I can use the binarization function on my cool image, and that looks like this. And here's the output. Or I can use my cool image in a manipulate. Or I can do some sort of color negation or segmentation or whatever it might be.

Be.

So you can refer to variables that you have Saved outside of chat cells, inside of chat cells, which can be very, very powerful. Instead of only being a conversational back and forth tool, you can really use it as a contextually aware tool. Let's see here, we got a couple. So Krasimir asks, does the new development from OpenAI regarding the extended inference time diminish or increase the value of a computational language plugin?

Huh.

I'd have to see exactly what the mechanics are behind that. I'm not sure if Alan has a feel for that.

I haven't had a chance to play around with the. Are we talking. We're Talking about the OH1 model, Strawberry, the thing that just came out yesterday.

Some people definitely brought that up in the chat as well. So I assume that's what, what we're talking about.

Yeah, yeah, I think that's right. I'm not sure yet. I mean I, I think that it's going to. For things that require more complicated reasoning, I would expect it to be much better. And they're certainly making the claim that it's going to improve math and other things. I've seen some people commenting just this morning. People have been testing this and there are still just fundamental errors that it makes in math because it just. That still isn't something that LLMs are, are good at or necessarily designed to.

Do and architectural issue with these things. So you.

Right.

I mean I could improve it, but yes, it's not going to replace something that is, you know, actually has these algorithms built in.

Right. You know, I am interested to play around with it certainly for more complicated kinds of projects and things that require more stages of reasoning and more complicated kinds of reasoning. But you know, this the difference between reasoning and calculation or computation. And still I'm not seeing that the, the calculation computation part is making a huge leap forward. I think there's, there's, it's great if this means that the model is going to be better at using tools intelligently and figuring out when and how to use them. Because that's something that we, we've seen problems with. So that's, that's what I'm interested.

Yeah, I think we look forward to testing that. Let's see here. So remble from YouTube asks and this is one I think I can handle. Not sure where to access this functionality like Wolfram and LLM API access. So if you have the appropriate OpenAI account, is it maybe a ChatGPT plus account? They keep changing the names. You can get an API key from that account and then in a, if you have you know Wolfram language open. All you have to do is go to, you know, Wolfram Settings and there's a whole page in the AI. There's like a whole page dedicated to AI settings where you can put in any API key for any of the services we support and then your chat notebooks will just work. There was a repeated question from Luis about how do you get the key for ChatGPT so you have to have a particular OpenAI account. Again, the name escapes me. Alan might know the current name.

Well, there's, like I was saying before, there's, there's different things. There is, you know, there's the OpenAI API and there's, there's the, you know, the Chat GPT subscriptions and the, if you have a. I think you're right, it's still ChatGPT Plus. If you have a ChatGPT plus subscription, that doesn't give you an API key. That, that just lets you access ChatGPT on the web. If you want to use ChatGPT with chat notebooks, you need an OpenAI API key. And that's, that is a, that is a separate subscription to pay for. That's, that's not part of the Chat GPT web subscription.

And to confirm kind of what we're saying here in response to another question we have from YouTube, which is to use chat notebooks, we need an API key from ChatGPT. It doesn't have to be from Chat GBT or OpenAI. We support other models. But if you do want to use ChatGPT specifically in a chat notebook, you will need an OpenAI API key. Yes.

Right. And just to, just to remind, like, there is this whole list of things that we've got the plumbing in place to just directly, very easily plug in an API key from some other service. And then that lets you choose between different model versions on the fly and all kinds of things.

And let's see if there's anything else in here. There's a question about whether in the future you think it would be possible for Wolfram to utilize local, smaller models just without having to connect to something online. I can't speak to future directions, but that certainly seems like a reasonable thing to want.

Yeah, no, I mean, I know that I think we're moving in that direction. Certainly, you know, seeing the, some of the latest open weights models from, from Meta and others that are getting closer to being runnable on a reasonably powerful personal machine. That's, that's definitely the direction that I think we, we see things going. We're not not quite there. Where. I don't think we're there where you can as easily as this plug in a local model. But there are ways to use the chat functionality in Wolfram with local models running with the Llama and other things. And I think there are articles on the Wolfram community and elsewhere that talk about how to set that up.

All right, that sounds good to me.

Let's see here.

Is there any option for Ollama or LM Studio support? So Llama, one of our supported models.

Olama is not. It's not really a model. Right. It's a framework for, for being able to work with local models. Again, that's one. If you. Search on Wolfram Community and there is, there are some. There are a handful of posts I've seen there with people and then some people who have posted notebooks that have a very easy way of hooking into a local Llama server. So whatever local models you're running, you can, you can access those.

Yeah, perhaps. I wonder if Anton Antonov has some posts about that. It seems likely to me.

Yeah, I think here was, Here was one. I know Mark o'. Deal. There was. Yeah, there's a notebook in here. There is a nice kind of easy get up and running with hooking into a local Ollama server here. I did this. It worked really well for me. And you can, whatever models are available through Ollama, you can install locally and play around with those in addition to all the commercial models that you can just bring an API key into chatbooks and run.

All right, well, that's excellent. Glad we have somebody, especially somebody as knowledgeable as Marco on Community to help us answer this one. And I think that that's all of the questions we have. So is there any last things you'd like to say about the integrations here, Alan, or have you pretty much said your piece?

I think I've said my piece. You know, it's. It's been startling but really exciting to see how things have developed over the last couple of years. I mean I still very clear that we remember, I guess, the end of what the end of 2022, beginning of 2023 and the first release of ChatGPT and those first explorations that I was showing early on with where we were saying, oh, this is really cool. You could copy and paste stuff from this into a notebook and run WolframAlpha code and then copy and paste that back into ChatGPT and get something out. And to be at this point now where I'm firing up a notebook and I'm chatting with an LLM and it's firing off queries and calls and writing code and pulling those back into a notebook. Just very, very cool and just opens up so many more possibilities for people who are, I think, new to Wolfram language and just trying to get a handle on what you can do with it. And even for people who are very experienced just streamlining some of their workflows and making it a lot easier to explore and prototype things and get to a point where you're sort of, you're improving code and playing around with ideas without spending tons of time with that initial kind of, you know, blank notebook, trying to start from scratch and debugging things. It's really exciting.

Yeah. I have to again, echo that sentiment. It's been pretty remarkable to see how far the integration has come and, you know, how useful it can be in a lot of cases.

Yep. So.

Well, we're at Time, so I'd like to thank Alan for the presentation. It was great to see some of these integrations we've been alluding to over the week. You know, get their time on screen here. And I'd like to thank everybody for attending. I hope that this was an instructive study group that kind of clued you into some of what's going on and, you know, what the future holds for this very popular but somewhat confusing new topic on the social consciousness. So you've all got your recordings. You'll get a recording for today as well. If you have any questions that arise while you're watching these or just something occurs to you, feel free to post about it on Community and we'll do our best to answer your questions there. So thanks again for your attendance, everybody, and I will pass it back to Cassidy to close us out for the week.

Thank you so much, Urban. And thank you again, Alan, for being here. It really was interesting to get to see all the, the new things that have been added in the last. Like you said, we started the first chat GPT study group, I think, in April of last year. So a lot has been done in the, in the time. Well, as we've mentioned and Arvind has said, we always encourage you to keep up with us on that Community thread. You know, go get that the Steven's book. Read through it. Sign up for the Exploring AI Tools with or Exploring AI Foundations with Wolfram Tools course sequence to kind of dive further. Arvin teaches two of those, I believe, as well. And yeah, we hope to see you back at our next study group. Get your quiz done by the 20th next Friday. And have a great weekend, everyone.