Using the Wolfram Plugin for ChatGPT: Live with the Dev Team

Okay, welcome everybody, to this live stream. My name is Alan Joyce. I'm the director of content development for Wolfram Alpha. Also work on a lot of other projects having to do with the knowledge base in Wolfram language, and have also been one of the people working on, behind the scenes on this plugin for ChatGPT, bringing some of the power of Wolfram Alpha and the Wolfram language to chatgpt. So I don't know if people hopefully have been following over the last couple of months, starting off the year with Steven's blog about some ideas for how someone might bring some of WolframAlpha's computational knowledge superpowers, as he put it, the ChatGPT. So, interesting blog and looking at some cases where I think, as most people are aware right now, that ChatGPT and other LLMs kind of come up short with things having to do with math, different kinds of computation, hallucinating facts about things when you ask questions. And so we were playing around sort of manually with ways that you might try to take the output from Chat GPT, go to Wolfram Alpha, go to the Wolfram language, feed some of that data manually back into Chat GPT and improve some of the results you get. Shortly after that, we were contacted by people from OpenAI and they invited us to work on developing this plugin for ChatGPT. Initially, we had been thinking it would maybe just be a wolframal plugin, and then realized there were some very cool possibilities if we had a combined Wolfram Alpha, Wolfram language, making use of the Wolfram Cloud plugin. And so we've been developing this at a very rapid pace over the last couple of months, and it's really great to see this finally come out into the world. I'm assuming that most people who are watching this probably don't have access yet, because the initial rollout from OpenAI and this is all controlled on their side. They're trying to be very careful and concerned about safety and the experience people have with this. But a small subset of people who are ChatGPT plus users have access over the coming weeks. Their plan is to expand this and roll it out to more plus users eventually, possibly beyond the plus users. Right now, certainly we're trying to give people who don't have access more of a sense of what they'll be able to do when they get access. And for people who do have access, maybe spark some other ideas about what's possible. And certainly we're interested to hear from anyone who does have access to the plugin with some things that you've been seeing, whether they're suggestions for improvements or just other kinds of cool ideas and experiences you've come up with. So we, one of the, one of the things that we realized early on was we have a lot of different APIs for WolframAlpha. We have spoken results, we have full results, realized pretty quickly that we could probably get better results if we designed a new endpoint that was really specifically for ChatGPT or other LLMs to call. So we have a new endpoint for WolframAlpha, still very much undergoing development, that is designed for an LLM to call out and get WolframAlpha results in a format that's easier for it to consume and parse and get information out of than the kind of fully structured API results. So that's been great that we've been able to do that quickly. We've also had people working on the Wolfram Cloud side and certainly with, you know, all the functionality of the cloud, been able to rapidly prototype and deploy a custom cloud API that also does some other kinds of things with the data that's coming back from cloud evaluations. And in the course of that, even sort of teaching ChatGPT how to more efficiently use the Wolfram language when it's not doing things quite the optimal way. So if people haven't seen how this works, certainly there are things that are possible here that are kind of obvious. Or when you do have access, what's going to change in the ChatGPT interface is you can choose from different models. One of the models that's going to become available when you have access to the plugins is this plugins model. It's an entirely different thing alongside of versions of GPT 3.5 and GPT 4. But if you choose the plugins model, then you have access to what OpenAI has started this plugin store. And so there are a number of different partners and, and there's a Wolfram one. And so you really just click install and then that will show up in your list of plugins. You can, you can add, I think, up to three plugins at a time. I'm just going to focus on the Wolfram one today. There are also instructions they want to encourage people to start developing more plugins for the system. And so there are ways for people to prototype those. But for now, you activate the Wolfram plugin and then you can just do, you know, we'll do sort of straightforward variable from alpha type things. You know, how much is 2 plus 2? What happens then is the model says, all right, you know what, I can call Wolfram for this and I'm going to get some data back from that API. Hopefully they're moving a little quicker than they were yesterday.

Yep.

And so what this has done is it's made a call out to just Wolfram here, but you can look at the result and see that this is data that's coming back from Wolfram Alpha. And so this is kind of representative of what the LLM API looks like. A very stripped down kind of WolframAlpha. This is your query, here's our interpretation, here's a result, and here are forms of all the entrained data that typically comes back in Wolfram Alpha result, including links to images. And you'll see that, you know, one of the things we've been trying to do is help teach ChatGPT how to pull images out of results that come back from us and include them in results. You know, we can do other things that are, again, I'm sticking with the very kind of pride and true Wolfram Alpha sort of things and there are a lot of other people on this call who will show more complex and interesting stuff. But again, you know, simple kind of entity property query calls out to WolframAlpha. There are probably a lot of people using it right now. So the system's a little slow. But again, digging in, getting the data back from WolframAlpha. And it's all sort of in this just, you know, standard tabular, simple form. But it's going to present it in, you know, a nice narrative summary of the information that we got back. It'll pull in charts in appropriate places, summarize some of the other data in the result. There are, you know, certainly you can also do things that are going to probably. This is going to call the Wolfram Cloud, although it's kind of a WolframAlpha E imported input too. But let's see what comes back. Yep, in this case, this said, okay, this is, this is doing a plot and so it's going to call out. It actually taking that, writing Wolfram language code, feeding that to the cloud and then getting it back. And we actually get a plot with a, you know, it's nicely labeled. You can build on some things like this and say things like, I don't give that plot a different theme and think it will know how to, how to add in an option for this too. And so it's going to make a call, it'll modify its previous call and yep, give us something with a different theme. From, from among the things that we support. When you're playing around with the plugin, sometimes the model makes a decision about calling Alpha or Cloud, and you can certainly encourage it to do just one or the other. So you get a result and you say, I'd actually like oop. And sometimes it tries doing things that don't get supported by Wolfram Alpha. And so in this case, you know, we didn't know how to handle the scientific theme, but it just went back and it tried again. And this is the interesting thing that we see a lot of times with the plugin is something fails and it'll say, okay, it didn't understand that. I'm going to try again and I'm going to simplify that input and it'll get something useful back. Certainly there's also things like here's an example of kind of how we're using the Wolfram language itself to try to teach Chat GPT a little better how to use it. If I just said, give me Joe Biden's height, it most likely would call Wolfram Alpha directly, because that's more of an Alpha kind of question. If I force it to use Wolfram language, what you're going to find is it just doesn't know precisely all of the canonical names for entities. And so what we're starting to do with the response from this API is say, you know, if you get this kind of error, here's a tip for what you can do to actually get what you're looking for. So we suggest that it goes and it uses Interpreter, it makes that call, it gets the right entity expression and then it feeds that into a correct call and gets that data back out. So some really interesting things that are possible here. I am going to keep my part this short and I am going to turn things over to other people who have a lot more interesting experiences through math chemistry, a lot of different domains and kind of showing off more of the range of what's possible with ChatGPT and Wolfram.

Fantastic. Thanks, Alan, for that great introduction. I am just going to start us off here with a new chat. My name is Jeremy, I'm one of the Wolfram Alpha math developers. And yeah, going to, going to walk through some more basic examples and sort of step into looking at how we can use ChatGPT to do things beyond just basic computations and really kind of push, push it to do some complicated math. So just starting things off, one of the, obviously the great thing with it is it's very conversational, so you can set it up by saying, let's do some math together. And yeah, it'll say, you know, something friendly. And I have been looking at a number of AP calculus problems and there are a number of things I'm going to start off with sort of a paired down version which this is the kind of thing that the, where the strength of an LLM really comes out because it's here we're giving a problem that has extra information and in this case, In this case, sorry, just double checking something in the background here. We're giving extra information that isn't relevant to the actual, to the actual question. And it's able to pull out just the necessary pieces. And I ran this just a few moments ago and it, and it provides a very verbose answer, but it starts by recognizing that for an area between curves problem, we have to find where those curves intersect. It sends a call to WolframAlpha, as Alan was showing before. And we're able to see both the query that we send in and the results that you would get on WolframAlpha. And then as this goes along, it's now able to put together an integral to compute here, which it's then also sending that off to Wolfram. Sometimes it does get stuck in the middle of things. So I'm gonna say, did you get stuck? And it's always very polite and says something like, I apologize. So here it's gonna give it another try and this time it looks like it's actually putting together the full Wolfram language call to the function integrate to be able to give us that exact answer. And let's see if I can say, can you be inexact result instead. So this is really great because you're able to conversationally work on your problems here and conversationally interact with something that is deterministic, like the Wolfram language systems, where you're able to really always have the exact answer that you want, but you know, also be able to have the fluidity of ChatGPT running. So, yeah, and so this is great. This was, this is the kind of thing that an LLM is not particularly good at is being able to calculate this math. So being able to make that call to Wolfram is really fantastic. I'm going to show another example that is a little bit more contextual, which this is an actual AP calculus problem. And again, there's a lot of, there's a lot of information and this is a pretty long prompt here, but we have. The problem is talking about fish entering and leaving a lake over a period of Time. We have a function for the rate of fish entering the lake, we have a function for fish leaving the lake. We have real times and we also have sort of the time scale that we're working with From T equals 0 to T equals 5. And as you can see it's. This is really flying along here. And it made two calls to the Wolfram API here. Here it looks like. Oh yeah, this is. It tried to construct some Wolfram language that didn't quite work. So it ran it again and this time used NF integrate to come up with an answer. And it was able to do that rounding to find the number of fish that entered the lake over a period of time just on its own, which is fantastic. Some more interesting examples that I've come across have been where you're able to sort of push the limits of what code it's able to create. So in this case I was working with just seeing how, how well it could go about making, do it doing things with matrices. And this is a, this is a fairly involved input here, making a 4 by 4 upper diagonal matrix with increasing primes in the non 0 entries. So if you do this it'll start thinking almost immediately using Wolfram and it'll, it sends this all in line. Whoops. Let's see, did it get it that time? Yes, there it goes. So it messed up the code the first time and it recognized actually that that was not quite right. So it tried again and did get something that is an actual matrix. And because of the non determinism of it, sometimes it displays this with latex, sometimes this, this is a new display, I haven't seen this one in a while. And then we can say something like great, now can you take the inverse of that matrix? And again recognizing that this is going to be a query that Wolfram can handle, were able to get that quite quickly. Can. Say I don't, I don't like the way that that's looking right now. Oh, this is not what I was expecting it to do, but.

I guess.

That is a little bit more nicely. Fantastic. So you can see it building the latex call the latex statement live for us and then this one, just to sort of push things even a little bit further, we can ask it to implement a fairly old and well known algorithm for finding prime numbers. So in this case it's called the Sieve of Eratosthenes and we can let this run and it's giving a description of what that algorithm is, which you know is a useful reminder if you're not remembering what that is off the bat. Whoop. Let's see. Okay, so it's going to try again because it did get something but it didn't, it didn't make a good call. So. And here it is having done that correctly. And now I'm even going to ask what was the code that you wrote to do this? And now it will in a nicer way with very nice comments explaining what's happening throughout this. Which in this case it might be nice to be able to do this without a for loop sometimes. I've seen it, I've seen it do things, do this without a for loop. But it did, it did work and it, yeah, correctly selected all the prime numbers. So this is just showing sort of the, the scope of what the, the combination of something that's non deterministic like Chat GPT and also the, you know, really fantastic computational powers that we have behind the Wolfram language and Wolfram Alpha products. And with that I think I will be handing things over to Brad. So I'm going to share.

Thanks Jeremy. I will start my share.

And.

There we go. Right, so my name is Brad James. I'm the WolframAlpha math content manager for Wolfram Alpha. And one of the things that I've been playing with ChatGPT, you know, our group, our company is always thinking about the future of education, math education. And I'm curious how LLM tech like ChatGPT can be used as a teaching or tutoring helper. So in addition to like the raw computation and explanation of process, like a key aspect of education is getting feedback on your thoughts and ideas and attempts. So playing around with some prompts, I've found variations of this to be helpful for this kind of conversation. So again, always very happy to help. So here I have a kind of classic math problem of two vehicles traveling in different directions and I've shared my initial thoughts on what, you know, how I might go about this and then playing a little bit ignorant as to how I would do the actual conversion there. So it seems to be setting up a nice system of equations here and I asked it not to solve for me, so it actually listened and respected that. So, you know, I have two equations, I have a couple of unknowns, or it looks like just one unknown, but let's just ask it to go ahead and.

Solve.

And right away it's calling Wolfram. So we'll see what, what kind of result we get here. Okay, so one of the speeds was said to be 24. Yeah, 24 miles per hour, 32 miles per hour. That sounds right. But, well, I don't know what the answer is. So here we're asking it to verify. We're kind of giving it a hint as to how it might go about doing that verification. And. It looks good. And then those will add up to. Yeah, sometimes it does the math by itself. If it's simple integer arithmetic, it's usually pretty good with that. And this is one that most people can probably do in their heads.

But.

All right, so let's try another problem kind of along these same lines, but maybe a bit more advanced. So here we have two salaries. One gets a larger raise.

Each year.

And one gets a smaller raise more frequently. And I shared my initial thoughts saying that it kind of feels like a trick question. And I like some of the kind of feedback that it gives here. Like, ooh, interesting problem.

Here.

It looks like it's just raw computing year over year, the values, but then it is actually plugging them into Wolfram. I've seen it do this problem with index summations before, which I think is a bit more interesting mathematically, but this is fine. All right, so it looks like my intuition on this problem was correct, that after at least a few years we're going to start getting a higher salary with the second one. So if I say something along those lines and ask for a further computation that it can do.

Right.

When would these two salaries be equal? And tries to do some other computation here. Not sure this is a correct equality because this is half of a year's salary. I'm not sure. And certainly getting a negative value here is also unexpected.

Well.

Maybe we can move on to the next problem. So sometimes, even though it is calling WolframAlpha, if it calls us with incorrect, you know, some sort of incorrect assumption or equation, we're only going to be able to solve the equation that we can solve. I'm sure I could provide feedback to that. And it has gotten this problem more correct in the past. But let's try one more problem here. So this is kind of geometry problem with potentially some, some real world information baked in. All right, so it knows some things. It knows that there's £2,000 in a ton. The density of iron is something that we can answer in Wolfram Tech. Sometimes it likes to call us for that and sometimes it, sometimes it just picks it from its own internal knowledge. But let's see here. This one, I did try and trick it up a little bit by suggesting an incorrect formula for the volume. It kind of just blew right Past that. I've had it before, kind of gently correct me on that, which is something that I would, would enjoy in a tutor.

Right.

You want, you want it to be able to reliably tell you when you did something wrong. It didn't do that in this case, but I'm sure you could construct a prompt that would make it more willing to do such things. And let's see if it gets the final result here. Oh, there we go. I'm curious what it failed to do. Oh, it had some, some strange variable names that WolframAlpha probably has a little bit of difficulty. Pars.

Well.

42. Yeah.

Okay.

So it did use some of those values. It got that from us. So anyways, interesting back and forth. I hope I was able to kind of show how it can be used a little bit more conversationally, I think. One other neat example, if you, if you have access to our plugin, you probably have access to some of the other plugins I was playing with word problems that would use multiple plugins to answer it. So here is one where hopefully it will look up the flight data from this Expedia plugin and then use us to kind of compute the result from there. And it really likes to tell you everything about the flight.

Right.

So there it calls Wolfram for the distance and that's not a bad flight price, actually. And then let's do the math. Beautiful. All right, well, I think that's probably about it for me and my time. I'm happy to hang out in the chat and try and answer some questions that anyone might have. But for now I will hand it over to Jason to talk about some fun chemistry stuff.

Thanks, Brad. Let me share my screen here. So I'd like to start with a blank chat. And since I'm doing chemistry and I'm a little particular about seeing my chemical formulas formatted correctly, I'm going to start off by asking Chat GP to do that and then letting it know that we're going to do some chemistry. So. Oh yeah, this time it decided to show that it was really paying attention and gave us some examples. So let's start with a. One of the common problems that that one has to do. Let's start with a reaction balancing.

Problem.

And again, this is something that you could push through Wolfram Alpha, but we'll see what it's actually going to do. Now, if you caught that it did its math wrong, which it often does with this particular problem. And so we can say check your work. Let's see what it's actually doing.

Yeah.

So it's putting together a requested added chemical equation which unfortunately may not, may not work. So it should try it again. Let's see if that. And that does work. So it should come back with the correct answer where it has H8 hydrogen chlorides that are necessary as reactants to balance this. And just to look back to see what happens is that it doesn't count the number of chlorides quite right. When it's balancing, it gets seven. Which is just a really nice example of why an LLM shouldn't be doing too much math. It should be calling out to WolframAlpha. So another example I'd like to try is just a simple unit conversion. These are at the core of many of the chemistry problems that are done in classrooms and research labs. So here we go. You might be wondering if I had prompted it to do the explanation of this in sort of a step by step manner. And the answer is no. This is all ChatGPT generating this on its own and being very verbose in how to work the problem. And it did. It got the right answer 3.011 times 10 to 22. And we can double check to see that it actually, that's actually the result from Wolfram that it got back rather than imagining things on its own. And indeed it is. So now we can, at least in the chemistry area it does pretty good with more complicated word problems as Jeremy was showing with math. And so let's try an ideal gas based problem. And you notice we had asked it to typeset chemical formulas but by now it's already forgotten. I'm sure if I remind it here it would would pick up doing that again. But this is the sort of things that sometimes happen and you can give it instructions that it chooses not to. And indeed the 655 Kelvins is the correct answer. And we can go back and see what it was doing here. It's passing the full calculation on to us and getting the result here. Yeah. So 655 Kelvins. So pretty impressive for how it can dissect the problem, come up with all the individual information and then put that over to solving the problem. Like to do another one that's a little bit more interesting. Here's a ph problem. So this is an equilibrium problem and depending on how it solves it, we might actually see an equilibrium table. Sometimes it chooses different paths of how to display the results. So it's worked out the problem balanced chemical equation. Again, it wasn't doing subscripts, but it set up the quadratic formula that it's going to need to solve. And now it's doing a calculation. Notice it's paying attention to units. And it came up with an answer of 1.47, which is incorrect. So let's go see what it, what it actually did. So up here it was solving, it was trying to solve that expression by passing the full in. It didn't quite get something that worked. And then let's scroll down and see what it tried over here. Ph equals negative log. That's, that's true. But let's see if it did actually got a result. But since it didn't solve the quadratic quite right, let's see what it does. Let it know that it was wrong. Hopefully that'll be enough of a prompt to push it forward to try again. Sets up the formula again. Check to see. Yeah, here it built a better input. This time it actually got the results. So it took the positive. It was smart enough to know that it took the positive value, not the negative one because we're talking about a physical quantity. And here we go. We landed again at the right, the right answer of 2.18. Let's see, let's do one more problem and then if time permits, we'll try a little something with coding. It's kind of a lead into the next round of examples. So here we're doing stoichiometry. This is, you know, we've got a reaction. We have so much stuff we're starting with. What are, what are we going to get? The first thing it did it now it balanced the equation on its own. I wouldn't always trust it with that. I would definitely have it double check. But in this case it got it correct. So as you can see, these problems are long and it's multi step. So it's doing rather well. Not only narrating what it's doing, but also doing the multiple calculations along the way. Yeah, 81.3 grams. That's the correct answer. So now I'd like to shift gears a little bit and show some of the coding that we can kind of do. Now the chemistry features in the language are always growing, so I'm going to be rather explicit with a prompt because I want it to go and basically read the latest documentation. So I'm giving it a little bit of code that'll allow it to pull in the documentation on the fly and then some other instructions to feed the prompt. So what it's going to do is hopefully go out and make that call and read in Quite a bit of documentation for the molecule, molecule Q and some other related functions and this usually goes quickly. So here it's going to give us back some of the details that it just consumed and the question is whether it will remember it. Okay, then I'm going to ask. What is the structure of the ATP molecule? So when I now got the 2D structure, which is what I was hoping it would do. Yep. Now plot that with the oh groups. This is the oxygen hydrogen groups here for those that aren't chemists in the audience highlighted. Here, it should have to go over. Yeah, it's doing a molecule plot and it's going to use a molecule pattern to highlight things. But those of us that are in the know, this oh pattern here isn't quite right. It's got its syntax wrong. That may not do. It's needed to for fix its molecule pattern. I'm going to let it go if it fixes it. Yeah. See the pattern it used only highlighted the oxygen. But let's see. In the interest of time, we'll go ahead and stop here and move on to the next set of examples.

I think it's me.

Yeah, I think so.

Okay. You should see my window. Okay, thanks. All my predecessors and co workers, you showed already a lot of nice stuff. There's not much left to show. So I don't have so much as a theme as my predecessors, but I want to show some random things that we did over the last weeks when we developed the whole functionality because it's really pretty cool what you can do. If you put ChatGPT together with Wolfram Alpha and Wolfram language, you can find poems, edit with math, get nice plots inside and the power of ChatGPT Understanding of language together with the power of Wolfram Alpha and Wolfram language for math and computation really allows you to do nice things like solve non trivial mass problems, do more complicated ones going back and forth, solve Fermi type problems. Okay, let's start with a slightly non trivial one that I copy because I'm a slow typo. I compute the arc length of the following three curves and plot the longest of the three curves. So it will go ahead and give me the general formula that it needs to calculate an arc length. And I want to do it for the three curves. And now it goes on. Now it will probably call Wolfram language, I assume sometimes it calls Wolfram Alpha, sometimes Wolfram language, sometimes it goes back and forth. See what it does. Today we call the Wolfram Alpha. This happens sometimes. I think it called too much at Once, I think now it will tie individually. Yes, of course. One time for first curve, one time for the second curve and one time for the third curve. Sometimes it works immediately, as we have seen already earlier, but it has the capability to decide what to do based on the result it gets if it fails. So it got, not unexpectedly, that cosine of 3x is the longest. And now it makes the call to get the actual plot. I think that's pretty nice. Great. So, in the interest of time, because Christopher is still after me, I will quickly go to some other examples. You could say something like make a log plot of this. And it's not particularly complicated, but it has to interpret it and it be able to provide perfect Wolfram language code which does this one with LCM applied to the range and makes immediately the curve say something like what fraction of all fractions of the form I J are already in lowest terms. It says what I have to do, I have to build them all. And then it counts and it gets the right answer. Here's a slightly non trivial math problem. Let F of X be a quadratic equation and assume that F of F of A is a certain number, what is a? So it does a mixture of something on its own, like the double substitution. It does it on its own and then it calls Wolfram alpha for solving the equation. And it gets the following four solutions, not unexpected because we have a biquadotic at this point. Here's a slightly more interesting one. Find a univariate real function whose derivative is the average of its second derivative and three times its first derivative squared, and it correctly understands this, writes down the corresponding differential equation. It brings it in a certain normal form which wouldn't have been needed. But fine. It's always nice to have the highest derivative on the left hand side, calls the dissolve function for work from language, gets the solution and then I can say please check, I'm not sure. And then it checks, it puts in both sides and indeed everything worked out fine. We have already seen it's very good with world problems. They can be quite elaborate. They can go over many lines. Like a man bought a number of turkeys as a cost of $60 and resolving 15, he thought rest. And it nicely analyzes everything, makes equations, sometimes one, sometimes two, depending what it needs, solves it with the corresponding domain, sometimes integers. And then you can always say like the last one, check. And it will give a nice proof that this is really a good solution. Similar one, a boy on being asked of his age, of himself and his sister Said three years ago I was seven times. And so on and so on. And again we see this works all beautiful. We get a solve and it gives the right answers and this works out similar one don't have to. And you can do more exotic things. Like you can say take three really nice integrals that you like or provided and then write a poem about the integrals. About the integrals. Enrich it with a picture and it will go on and it will write a poem about the integral of sine or the exponential. It's quite amusing what you can do. You can easily translate pseudocode that you find somewhere in an algorithm textbook like here one for the extended GCD as you find it in some lecture notes. And you can say translate is please import from language will do a nice job. Add comments that weren't even there before. Then you can say please run it and it runs it and it returns the result in the form of an association. Which is very nice. It's quite useful. Here's an interesting one. Distorting numbers of the second kind have the following representation. It's a product of n minus m terms where each of the K I have additional conditions. That's not totally trivial to do. Most people wouldn't immediately know how to write it, but it comes up with a pretty decent code. The code works correctly. It is not the most complexity theoretic best one, but it works. And you can say n = 8m + 3. It calls the code from above and it gets correctly. S83 is 966. Here's an interesting one. What's the largest disk you can put between the two curves? Cosine squared and minus cosine squared. And it says oh, I need kind of how much space do I have? And it says okay, There's a distance of 2 at x equals 0.

So.

The result is a dist of radius 1. This turns out to be wrong. Why? Because there are points that are nearer to the origin that are a little bit away from X equals zero and you give it a hint and say maybe consider a nearest point. And then oh yes, I should do this. And then it actually gets the correct distance which is a little bit smaller, which is only 0.9. I find it quite amazing that it can do this. Another one, there are eight poodles in a little together they weigh a certain amount. The heaviest one is 30 ounces and they form a geometric progression. And again everything encodes it correctly solves it. And sometimes in long deviations you must say continue because it ends here's a really nice one. I like this one that's a little bit longer. There's this famous Helios Dunkirk problem. We have this complicated puzzle about how many steels and cows and it works absolutely beautiful. It gets all equations and then it solves it, and it solves it all the integers because obviously with intro solution gets a parameter sized solution, then select the smallest one and yeah, I think I should end to give Christopher some time.

All right, thanks. And let me see here. Right, so yeah, so I'm just going to show a couple short examples.

So.

Right.

So one thing that actually you saw, maybe especially in some of Jason's examples, is how you can sort of initialize the conversation with a kind of prompting message that will change the behavior of the chat for the rest of the conversation. So he had some examples of either prompting it with here's some Wolfram language functions that you should use in later responses and things like that. Here's an example maybe that I was writing earlier. So basically this is just telling it that it should always produce a relevant Image Using either WolframAlpha or Wolfram language, you know, based on whatever I ask it to do. And so Michael, by the way, I think your microphone is still on. But yeah, so let's try maybe an example. I'm not sure what this image of a smiley it thinks it's trying to display here is, but it didn't quite work. But okay, so we can try, we can try something like maybe tell me about koalas or something like that and if it can write this hopefully a bit faster. Yeah, let's see. You know, it writes a very standard kind of ChatGPT style essay about koalas and then, oh, this did something interesting. That's something we've tried to prompt it not to do, but it seems it's done it anyway and that is make up fake links to what it thinks are images. So essentially the problem here is that when it displays an Image in the ChatGPT interface, the way it does that is by writing markdown containing a link like a URL to an image, normally one that was returned by something like WolframAlpha. So in this case it just in chatgpt style made up a fake link that obviously doesn't work to what it thinks is an image of a koala. So let me, let me kind of see if I can get it back on track. I'll just tell it you didn't actually make a request to Wolfram to get that picture, did you? And maybe we'll See if that convinces it. Yes. Okay. To write the error in its ways and. Right. Of course it apologizes. And there we go. There's a beautiful image of a little baby koala. Maybe let's try one other example. So I could say something like, tell me about Chicago, and we'll see if this. Well, first, actually, it's doing something nice, which is it's asking WolframAlpha for information about Chicago, which it can then use to incorporate into. Oh, I guess this made a bullet list. Often it will incorporate into a little essay about Chicago. And then, of course, it'll include an image, because that's what we've told it to do. Maybe one last example of this image prompt would be something maybe more mathy. So maybe I'd ask it something really easy, like where does Y equals x squared -2 intersect the x axis?

Right.

So this is pretty easy. But it'll go off and give it to WolframAlpha, by the looks of it. And hopefully, because I asked it, yes, it will always include an image. And so we get a little graphic showing where this curve intersects the X axis. So that's nice. So I think it's kind of an interesting thing to play around with these starter prompts because you can really change its behavior quite a bit and make it do some interesting things. So one other example I'd like to show, which is one that I think the Wolfram language prompt has made particularly possible, is one that sort of makes it changes the flow by which it programs. So essentially I have this prompt here. It's a bit longer than the previous one, but essentially what it says is it kind of just outlines a pretty standard programming workflow. So it tells it, before you write your function, I want you to write a couple of test cases, and then I want you to write the function. Then I want you to run your function on each of the test cases. If any of them fail, try rewriting your function and sort of repeat until it works. So essentially, it's just outlining a debugging workflow so that ideally it can produce more accurate code. So if I run this, it should just tell me, yes, it understands what to do.

Right.

It's just repeating basically what I told it. So now let's try giving an actual task. So I don't know, maybe a simple example would be something like this. So write a function for testing whether a string contains a US phone number. And let's see what it does. Sometimes this works, sometimes it struggles a bit. But let's see. Okay, so. So you see, it's written a few formats that it needs to support. Okay. It's written what it thinks is a program for, for doing it. And it's going to run that on some test cases. Ok, you see that it, the test didn't pass, so it's now revising the.

Program.

And we'll see it continue to go through. I think this is a really interesting use case though, because. Right. So programming is a, you know, is a common task that people have sort of been interested in getting LLMs to do and they're, they're surprisingly good at it. But, but at some level, you know, the way that people ask LLMs to program is really quite unfair. Right. If you asked a human, even a very experienced programmer, to write some fairly complicated program just, you know, on a piece of paper without being able to run it, this is a really hard task. Like that's a, that's a difficult thing to do. And realistically, no human programmers have to do anything that hard. They have a much easier experience of being able to program in an actual environment where they can run their code, see if it worked and so on. And so I think with a flow like this, you can actually kind of reproduce that environment. So, okay, so as I think this often does, it's getting really confused because it doesn't know how to write these regular expressions. Right. Just keeps looping, trying harder and harder with different regular expressions and it's failing. So, you know, I'm going to, to give it a hint. I'm going to tell it, maybe try using string patterns instead of regular express. So let's see, let's see what this does. Ok, so now it's rewriting it instead using string patterns, using digit character and white space character, et cetera. And now let's see how it does. So it's going to be running its test case. So now it's running it on that string there. Oh, that still didn't work. Let's see if we can revise it. This is, you know, this workflow is always a little complicated. So we'll see if it actually manages to, to work its way through it all. Okay, so that test now passed. Now it's running the next test. Oh, okay. It correctly identified that it forgot to include the function definition in that call. So that's good that it remembered too. That's one of the things that, for example, I had to include in the prompt is to tell it never forget to include the function definition because it always forgets otherwise. You can see that one passed 2 and this one was right, that was a valid phone number and this one was not a phone number and it returned false. And so you can see it all worked and it can tell us it passed all three test cases. And here's the final function. And I could even ask it just because it wasn't shown too clearly earlier. I could say maybe what were the tests that you run and did they pass? And maybe it can get it to give me a little report about the tests. Yeah, here we go. Beautiful. I have found that sometimes you have to be a little careful because for example, when it's running test, sometimes it lies to, it'll say something like the test returned the wrong answer and it will say oh, the test worked, but it clearly didn't. So you do have to prompt it quite carefully to really nail home that it has to take the test seriously. But yeah, so I think this is a, you know, I think this is the kind of workflows I think are really interesting because it means that it, it can take advantage of this sort of back and forth interaction with the plugin that's otherwise really hard to, to get and which means that the LLMs can do a lot of things that they couldn't do otherwise. So yeah, so I think that's, that's about it for my examples. But I think we're maybe going to try and answer some questions now. So if any other panelists as well would like to answer any questions then please just jump in if you see any questions because I've been sort of distracted for the last few minutes. But yeah, yeah.

I know there have been, there's been a fair amount of discussion in the chat itself and I think some responses have been coming out as we've been going. So thanks to everybody who has been engaging with us in the chat and been, yeah, sharing, particularly sharing the excitement. It is an exciting thing that we're, we're looking at here as I'm talking. I don't know if anybody else had a, had a particular question that they wanted to respond to.

So maybe one, one question was about can we use Wolfram and LangChain and how does it compare?

So.

I don't have too much experience with LangChain. My impression is that they, that LangChain has had for a while now a Wolfram Alpha. I don't know what they call it, essentially an interface to WolframAlpha, which I know that I think I've seen some of their examples and some of their tutorials and things that they did on their own. I don't think we had much to do with it, but I think we've certainly been interested in that and we've been sort of been watching it a bit and you know. Yeah, yeah, it's quite an interesting area. I don't know exactly how LangChain relates to this new plugin interface. My impression is that they kind of work in similar ways except that the plugin interface has a little bit more the model that OpenAI is providing is a little bit more specialized to this kind of plugin calling flow, but I'm not sure exactly what impact that has on its performance.

Few questions on like logic games and puzzles and things like that. I don't have any answer, but I also want to explore some of that. That sounds neat.

By the way, whoever was just talking with Mike was very soft.

Sorry about that. I think that was me. I was just mentioning the questions about puzzles and games sound sound like fun things to try. I'm looking forward to doing that. I don't have any examples or answers to those questions, but.

I was answering a bunch of questions on chat as we went through. I can try to sort of summarize those quickly. I guess somebody was asking about is this using Pro version of WolfRamlph or the free version? There's no step by step support or other Pro features via ChatGPT yet, but it's something we have been looking at. If you do get access to the plugin through ChatGPT, there's no Wolfram Alpha or Wolfram language subscription that's required right now to use the plugin. As long as you have access to the model through OpenAI, you'll be able to access everything that's available in the plugin. Somebody else was asking about time sensitive data and it's definitely not just limited to math, geo and things. It's the API the plugin is calling has access to almost all the Data available on wolframalpha.com it's only a very small number of licensed data sources that can't be passed out this way and so the data will be as up to date as the WolframAlpha website itself. I think there was a question from Scott Miller about interaction between Is there a lot of interaction between Chat GTP and Wolf, VPT and wolframalpha for instance? Often Alpha doesn't understand input. Does chatgpt keep trying until it gets useful feedback and hide the interaction from the user? ChatGPT's interaction with sources and its plugins are never hidden from the user. You can see all Those little gray boxes that show up. Each one is a call to Wolfram Alpha or the cloud. And you know, you've seen here you'll often see multiple calls just within a single answer or calls back and forth between each service. You know, keep trying, trying a number of times until it gets what it wants or it says, you know, I need to give up at this point.

Just as an example of that, with the balancing case that I showed earlier, all it needed to do was ask for balance and then the chemical equation. But it put in extra verbiage in there that didn't quite parse at this time. And so there are times when you know, the, the LLM will add stuff that it doesn't need to. And if you remember and look back up in the recording, it did try a couple times to modify its input. So it does do.

All right, well I think that's. Can I share things please? Yes, let me.

With respect to the puzzle, I actually tried various and a simple puzzle. It's quite good. So I thought I. The most complicated of all the so called Einstein's riddle. There are five houses and five nationalities and it does do a pretty decent job doing this. And sometimes it calls satisfiability, sometimes it calculates all possibility. So it's not perfect but it does pretty well with them. So I encourage everybody to try it out when they have it on their own with various prompt to nudge it into one or another solution direction.

Also incredible. The code it was just writing was using the function through, which is a really obscure function. I'm amazed it knows what through is. But yeah, I'm always, I'm always surprised by some of the Wolfram language code that knows.

You should try playing in some of the chemistry area. It is very creative with code that doesn't exist.

Right. Right through is a very old function though. So it's like version two or something like that. So it's had a long time to learn about it. It's just that it's very obscure. It does. You're right though. It's a different problem that often it struggles with like mostly new functions and new functions in kind of domain areas where it's. It's. Yeah, it both hasn't had a lot of time to see them especially since you know, what, what is it? September of 2021 or whenever their training cutoff is and. Yeah, yeah.

All right.

Well unless anyone has anyone else to add at the end, I think that's. That's probably about it. So maybe thank everyone for joining and for you know, also for showing examples and so on. So, yeah, I think that's about it.

Sat.