Injecting Reliability into Generative AI

So today what I want to talk about is CHAT GPT and how that fits in with the Wolfram worldview. But what today's talk is is fundamentally about the differences between statistical AI and symbolic AI. First of all, let me try and define the difference between statistical AI and symbolic AI. The large language models, ChatGPT and Bard and the whole collection of these things that are appearing at the moment are in this space of statistical AI, in that while they have this sort of appearance of intelligence, fundamentally they're doing one very simple thing, which is predicting the next word that follows some previous words. And so it's an entirely kind of probabilistic guessing exercise. But it turns out, and as we've discovered in the last year, as these things have emerged into the mainstream, that prediction can be incredibly good. And it's so good at using both kind of short range and long range context to use that to find that next predicted word and the word that follows that, that the words that it can predict often form the answer to a question. Or if the words that precede it are an explanation statement, the words that follow could be an explanation. Or if the words that precede are some instructions, it seems capable of following instructions. And this is where it starts. You get this emergent behavior that appears to be intelligence. But it's worth remembering that in the end it's guessing words using probabilities, because that does affect what it really can deep down do and what it can't. Now, Wolfram for 35 years, and me personally at Wolfram for 32 of those, have been very much concerned with symbolic AI. And I define that as having defined rules of behavior for structured inputs. So structured inputs might be something simple like two plus two, where we have an expression and the rules of behavior in that case are addition and, and there's no guesswork. We know exactly what the rules are for two plus two, that should equal four. And those rules scale up to all kinds of arithmetic. But they can also be more complex structured inputs like an image. And the rules behavior might be finding the edges of the contents of an image or they might be blurring an image. So these defined rules behavior tend to be quite deterministic and very, very, it's very easy to describe if you understand details, exactly what they're going to do. They're quite predictable. Now, of course, in our world, statistical AI and you know, used to be called AI when I started, really is come to be thought of much more as computation. And so these collections of deterministic defined rules of behavior cover all kinds of different areas, like things that are more modern, like machine learning, things a little bit older, like classical statistics. Structured inputs can be things like geographic information or graphs and networks and social networks. We can do operations on time series. Some of the operations might be things like finding the optimal values of things. The data types are quite varied between audio and video and text analysis. And there's all kinds of very specialized domains of computation in the scope of science and, and engineering and kind of systematic modeling. So it's a very big area that's taken out some of it hundreds of years to build up, but suddenly a lot of it's taken at least 50 years to build this collection of computation. So here's the interesting thing when you put those two worlds together, when you consider the pros and cons of statistical versus probabilistic versus symbolic AI, is that their strengths and weaknesses are almost the complete opposite. And this table really kind of sums up where I'm going to go in the next few minutes of this presentation. That the computational world, the maths and algorithms world, that's really great at things that are logical, that follow complex rules. It's proven very powerful in data science and it's very deterministic. If it works once, it'll almost certainly work again the next time. The large language model, the statistical AI, is actually pretty poor at some of those things and I'll show some examples in a minute. But where its strength lies are being very flexible and being able to handle a wide range of inputs that might be unpredictable. And the thing that's wowed everyone of course, is that it's brilliant of being human, like in its interaction. And it can handle poorly defined fields and unstructured information much better than, than kind of rule based approaches. So let's look at some examples to see what I mean. Here we look first of all at the weaknesses of the large language models. I said that it was weak at computational thinking. Well, here's a kind of logic problem that I've put into ChatGPT. All of my examples here I think are pretty much chatGPT 3.5, but these are fundamental issues. So while GPT 4 is substantially better than 3.53, fundamentally it has the same weaknesses, just hidden slightly better. So in this word problem I'm providing some information. If Alice is older than Bob, Bob was older than Charlie, Charlie's younger than Derek. And then I'm asking a question, is Alice older than Derek? Now the large language model writes a very nice essay here and describes how the relationship can be thought of as a chain and how it joins up and answers the question yes, Alice is older than. Then it follows that Alice is older than Derek. Actually it's wrong because it's got muddled in the logic just like humans do, if you don't think about it deeply. And in a computational world we can call that quantifier elimination problem. And we can show that it's not true to claim that for all possible Alice's and Derrick's and that set up Alice is great and Derrick. And in fact, if I reverse that, it's also not true to say the opposite. So it's an indeterminate problem. There is insufficient information here to decide who is going to be taller out of Alice and Derek. But the, the, the large language model, for all its ability to fluently answer my question, has got muddled in that logic because it's already too complicated for it to work through. Now that gets much worse when you get into algorithmic processes where you want it to follow a sequence of steps in the way that programming algorithms in a programming language are very straightforward to do. And that even goes as far as something completely basic, which is arithmetic, that I've asked it here, this problem, and if you chat to chatgpt or their ilk, then ask it 3 or 4 digit problems, it pretty much gets the right answer. But when things get slightly more complicated beyond things that it's seen and it actually has to start applying the rules, we can see that the answer we've got here is actually different from the answer that it stated. It's actually very close. It's kind of pattern recognition has thrown all of the right characters down, except that it's about 0.1% out. Because this digit here is different than the one that it imagined. And that's because it doesn't actually fundamentally understand arithmetic. It, it's really using recall to see problems that it's seen before. In fact, let's go to ChatGPT for a moment and do something only slightly more complicated. Let's do three to the power 30 and ask ChatGPT to give me an answer. And now we're into sort of much less typical computations and it looks like I'm not going to get any CPU time. Okay, 3 to the power of 30 is. Let's see how it does on this. 205,891. So let's compare that to the true answer. 2059 seems to have done a decent job here. Is it correct all the way through? So in fact actually on this occasion, it's got the right answer, but it's not something that can be relied upon when things get at all messy. Now, of course, when things get at all complicated, this breaks down completely. So here is its answer to a symbolic integration problem. And I liken this to an old comedy act in the UK where somebody's playing a terrible piece of music and he's accused of playing all the wrong notes and he jokes, now I'm playing all the right notes, just not necessarily in the right order. Which really, I think sums up the answer that it gives here compared to the true answer that we get out of Wolfram Alpha, that all of the bits in his answer look plausible. It's got signs and it's got causes, it's got a quarter in here that seems relevant, but this is fundamentally a different expression and is just the wrong answer. But it looks plausible because it's all the right kind of words in the answer. And that really sums up the way large language models work is they're focused on plausibility. Another thing that is a little surprising is that they are. They are no good with big data, which seems odd when we know that they've been trained on masses of data. And the models are extremely complicated and have millions or billions of parameters that have been fine tuned to create these models. But actually their attention span, the amount of information they can hold in a problem that they're now considering, is quite short. In GPT 3.5, it's something like 2,000 tokens. In 4, 4, it's something in the range of 20 or 30,000 tokens. There are models aren't quite so proficient that are doing 100,000 tokens, but that's still tiny when we're talking about data science. So here I've been slightly mean to 3.5 and I've contrived this question that I said, here's a list of numbers. And it's actually, when I typed this into GPT, it was more than a page of inputs I've omitted. So I can fit it on screen here for the presentation. 2000 items. So I've deliberately chosen a list that's just a bit too long for its memory span. And on the case that I ran this, it says the question appears to have been truncated, so it's forgotten the beginning of the question by the time it got to the end. In fact, now they seem to have made some change. When I ran this in a rehearsal for something else last night, it now deals with the truncation by somehow compressing the question. So it produces now a wrong answer and it just seems to pick a random number out of the list and it can't fundamentally deal with these large amounts of information because it can't hold it all in its kind of working memory. Just like you as a human. If I said read this list and then afterwards said what's the 40th item in it? You probably wouldn't be able to recall that. But you might have a working memory of 10 items. And I say what's the third one you probably could recall? The LLMs have the same kind of problem. Of course, in a computational world I can make a random 10 million length vector and ask what's the last element of that same kind of question, but many times larger? And it's utterly trivial in a, in a symbolic area world. But the one I think that most people, that people find most. Worrying is the reliability. And here's the example I like to show that I've pre captured on, on this occasion here where I've asked ChatGPT provide the data on livestock populations in Turkey. And it's given me a very fluent answer and it's given me some numbers. I've highlighted one of them that says that there were 33.4 million sheep in Turkey in 2020. But I've also highlighted something else here that I consider actually more dangerous than the fact that the answer is wrong here, is that it's trying to convince me of the answer by making it as plausible as possible by citing sources. It's been known to make up citations completely on this occasion. This citation is a real thing and in fact a well chosen relevant citation. So I, I went off and went to the Turkstat website and I spent a while going through Turkstat and found the relevant page that answers this question. And you can see that the answer is different, a similar order of magnitude, 40 million instead of 33 million. But it's just imagined the numbers and then tried to convince me that it's right by wrapping it in things like citations. So whenever there is knowledge coming out of the LLM or something that requires computing by non trivial reasoning, you can't trust the values that you get back or the facts that you get back. And that makes it extremely dangerous. Now, I've started by being somewhat negative about large language models. They are actually incredibly amazing and a great step forward in our technical capacity to deal with knowledge. So let's talk a few minutes now about what they're good at, which computation has always struggled with. One thing is this flexibility so we've had WolframAlpha for a decade or so and it takes natural language queries and it answers questions from them. But the way that we've done that is using technologies called Context free grammar, where we're having to pattern match the sentences using rules built by humans now for fairly simple straightforward queries on things that we know about. It does a pretty good job. But asking the question like what national land has a total humanoid population in excess of a billion has just too many distracting words for our parser to be able to handle that. You've got a map humanoid, you've got deal with words like excess. Is that a synonym for greater than a national land? That's a country. So there's all kinds of things I've thrown in to confuse things and we would simply choke on that. And you would have to learn to ask the question in a better way. The LLM is very good at taking that and rephrasing that into a short form, simpler version. That in fact is a question that we could answer with Wolf Mouse. So this ability to catch the messiness of human language and extract the essence is a really powerful capability that is certainly going to make Wolfram Alpha much easier to use when we get that properly plumbed in. And somebody was pointing out here, did I really make that mistake? I did have the Wolfram plugin enabled, which is possibly why this got the right answer. I think it wasn't actually thanks Sassness for pointing that out, but I think in fact in this case you actually got it right the previous time I tried it was hopeless but normally you would see used Wolfram plugin if it had, if it had used that. So I think in this occasion I can't claim that I just messed up the example. It did just get the thing right. Now you've probably seen loads of examples here, but for those who the concept is new here let's point out the the level of human like interaction that you can get that it can in simple prompts answer questions and write in particular styles. So I can ask it something like describe wolf mouth in a language suitable for a seven year old. And it already knows lots about the real world. So it doesn't need telling any More that what WolframAlpha is, it knows about that and it knows how to speak to a 7 year old. And so we get this very simply worded explanation of what it is you can use it for rewriting and extrapolating on. So here I've got a very long essay, something that probably I wrote on some previous occasion. And I've said summarize that in 50 words and it can take that and extract the essence and hit the target word count and capture meaning. It can go the opposite direction. So you can use it for very prompted argument making Giving it the simple argument that messy data is is difficult for insights and computable data can be used better. And I said write a 100 word paragraph that says that elegantly and we've got text synthesized around an argument that I fed. Can do things like write poetry, it can write song lyrics, it can suggest titles for things. It can come up with plots for a film and then write the screenplay for it with with dialogue. All of these things are really quite impressive in really being able to pass the Turing test. Now for the first time that you're not quite sure if you didn't know whether you're talking to a human or to a computer. Something they do a reasonable job at is things where we don't really know what it is that we want or how to do makes a reasonable job. And I've picked an extreme example here of something where probably no one really knows how you should train a dog to sing. But computation has nothing to say about this. If I was given this as a as a consulting project I would envisage a pretty big budget and lots of sample data for dogs vocalizations and maybe a physics model or who knows what how I might go about that Programmatically and logically. The LLM just has a stab at coming up with an idea and things that are in the middle where you kind of know what you want, you know how to say what you want but you don't know how to describe the rules for getting there. It does quite a decent job at and I'll show examples of that in a minute. Now this one I've got a very simple example and it seems very unprofound. But actually I'll revisit this in a few minutes. I think is actually one of the the most misunderstood powers of large language models is just a general ability to deal with structured and unstructured data equivalently to restructure, to extract facts out of poorly structured information to fix structure. It really understands structure. And so in this example I've given it a deliberately non trivial sentence that I want to extract a single fact from. And I'm saying how many dogs are mentioned in the following text? Return only the final number. So I'm giving it some instructions of what I want to return achieve. And the sentence is two basset hounds are walking down the street when they saw a Persian cat talking to a Labrador. So in order to get that number three, it had to process this sentence at the level of understanding the basset hounds are dogs, Labradors are dogs, but a Persian cat isn't a dog. And that the word two is associated with the basset hounds and that there is only a single Labrador mentioned. And so it can do kind of two plus one and extract the number three. And that is incredibly powerful capability, albeit in there a trivial example. So that's the background that we're. We're coming from. What we want to do our sort of outlook on, on AI at the moment and where we're putting all of our efforts is to try and take that best of both worlds. If I go back to my table here, we want to have a combination of computation and large language model that ticks all of the boxes. And what we've been thinking about and working on are the mechanisms to try and bring those two worlds together to make something that is the best of both. And if you've seen our public face on this, there's the Wolfram plugins ChatGPT which if we go back to GPT here and do the query that I did earlier, let's just make this win a little bit bigger. So I had. Tell me about livestock populations in Turkey and hopefully if this does the right thing, what we're going to see is that it's going to. Because I've got the plugin enabled at the top, the Wolfram plugin, along with another one which I'll show in a moment. It's recognized that we might have something useful to say about this and oh well, that's. Let's do a regenerate that. We actually let me do a new chat here and try that again. I think maybe the chat timed out in some way. At least I can say on that occasion we did the right thing and it was the LLM that did the wrong thing. But let's see if it does the the right thing if I just start afresh here. So hopefully again it will recognize that this is a thing that we might know something about that could be useful. And it seems like Chat GPT is being a little bit slow today. Okay. It's using Wolfram, so it's giving us a call and saying it's going to rephrase the question here into something we can pass. Actually, we probably could largely cover this sentence already. And now it's got an answer because behind the scenes it's had this conversation where it's used an English to English conversation to talk to WolframAlpha. WolframAlpha has provided factual information and then it has used that factual information in order to answer the question and write its fluent narrative. So because it got the data from us, it's been freed from this hallucination problem and it's got the correct number. Now, the 42.1 million sheep and all of the other numbers are correct because they came from a factual source. And it was using it large language model intelligence to decide that we could help with that and how to ask us. So just like, you know, I often liken the large language model to being that sort of super confident loudmouth guy you meet in the pub who thinks he's an expert on everything and has an answer for everything. While that person might spout nonsense at you to look clever, if you've got somebody next to them whispering facts in their ear, they tend to use those because they know that using reliable information makes them look better than ignoring it. But they only make stuff up when they don't know. So the facts fixes the reliability on that question. So that's the basic idea. It's reliability from computation together with the fluency of large language model. But when you start thinking it through, there's a few different permutations for how that works. So that is just to give a sense of how I'm breaking this down here. I was using the ChatGPT interface, so I've got a particular interface choice. And in this case the large language model was in charge. I'm talking to the LLM. It was the LLMs decision to give us a call and say, do you know the answer to this? And we can give it multiple points to contact for different kinds of information, but it's in charge of whether not to use those. So one problem is that it may choose not to. It may. If it doesn't think that the endpoint can help, it might go ahead and hallucinate anyway. So you have to try and manage that to make sure that you really convince it that it should. The kinds of things that should use the reliable endpoint for another way around is to put the computational, the symbolic world in charge and make the large language model subservient to that in order to provide LLM kinds of things they can do. So here's an example, doing the other way around. I'm in Wolfram language now and I've got this wrapper in our language that is our way of abstracting away the large Language model behind the scenes. At the moment we only support ChatGPT 3.5, 3.5 Turbo and GPT 4. But hopefully over the coming year, this LLM function, it'll be an option to say use Bard or use Falcon or all of these other LLMs that are emerging at the moment. And so the idea is this is an abstraction that says you will execute the following as if it was a function, but it'll be executed on a large language model. And here my function is not described in terms of programming code, it's described in terms of words. So my finance story function is write a blah summary of blah share price yesterday given the following facts, blah. And that's now. And maybe we should try and do this live. I run this right now. What it will do is randomly choose a style. Well, if I was doing this as a proper piece of computational journalism, I would compute that based on some forward projection. But here I've just randomly picked a style. I fetched some factual data out of our financial data feeds. So this is the Microsoft open high low close volume data. And I've labeled that in a way that makes it easy for the LLM to understand which bit of information relates to which. And I fed that into the slots of this LLM function. So the function there is, gets the script right and exciting whatever it is. Story about Microsoft share price yesterday given the following data and here's the essay that it's come up with. So now there's almost no chance that the LLM won't use the numbers because I injected them right at the start. He didn't have to make any decisions about fetching the numbers. I just said I want the following text with the following numbers in it. And I've so far never seen it fail to use information when you inject it in. It sometimes chooses not to. If you give lots of information, it doesn't use every piece of it and it decides it might be implausible. It'll say, well, it opened on this price and closed on this and doesn't bother to mention the rest. But I've never seen it hallucinate when you push factual information into the prompt itself. So here's the essay that's written and I guess looking at the writing here, that it's, it's chosen to be a pessimistic story. So we've got chat in charge, we've got computation in charge. There's a sort of two ways round of, of control. But we also want to think about what is the, what is that reliability that we're injecting. You know, I've. In the, in the plugin that we have publicly. Let's go back to that chat conversation that we had. Where is it? Is that one. No, it's not. Where is that one? There, there's two modes for our very generic plugin. It's trying to just do any as general stuff as it can. So one mode is that it asks Wolfmouth for a question and the other mode is, let's say, something like bar chart, that hopefully it will recognize the other mode of the plugin which is to synthesize Wolfram language code. Okay. It has not done the right thing. So it's failed to use the endpoint and said I had no idea how to do that. So let's give it a little bit more assistance since I use the plugin and hopefully this time it will do something more sensible using Wolfram again. And if we look behind the scenes, what it's doing this time, yeah, it's doing the right thing this time is it's synthesizing Wolfram language code, in this case the command that you would have to type in order to generate a bar chart. So a pretty trivial bit of code, but it's gone a bit further and it's adding some styling information and some plot labels maybe. Yes. And what it's got back from us executing that code. So we've taken that live synthesized code and executed this in a sandbox in order to generate a picture. And we fed the picture URL back to the LLM which has then embedded it into the. Into the answer. You have to sandbox that because this idea of being able to synthesize code is potentially dangerous because it might get confused that the right code for a bar chart is to do file delete, recursive and just try and destroy the machine at the far end. So we have to run that in a safe environment. But the idea here is that you now have very open ended kinds of things that you could do that if you can describe it within code. So those are both very general, they're open ended. WulframAlpha can do lots of things. Wolfram language can do lots of things in only a few lines of code. But what we also want to do is to encode our own knowledge or our own data into, into the system. So here's one way we do that. This is using the Chat GPT still as an interface. This is a kind of just a workflow utility we built called Chat Plugin Cloud Deploy. And the idea is this is creates the pipeline of creating custom endpoints for the Chat GPT interface. And it takes a few fairly straightforward arguments. The tool has a name, in this case UK Income Tax Calculator. It's got one or more endpoints which are the actual places it can call for help. One of the endpoints, in fact the only one in this tool, is called taxdue. And then it has a Wolfram language program that will execute on that API. It's an API that takes an income and expects it to be an integer. That's probably not really the right way to write it. It probably should just say number here because you might have something 0.50 on your income. And then in this case it does arithmetic. I picked this example, I was talking to some tax people. It's a quite nice example that sort of underlines the problem with LLMs is that the ChatGPT knows the main basic tax rules for UK income tax. But a few prime ministers ago or a rule was introduced to increase the top rate of tax without looking like it was increasing at all. Top rate of tax and the way that the that government did it was they took the tax free allowance, the bit you earn before you start paying tax, and made it go down as your earning went up. So for every pound above £100,000 in the UK, every two pounds rather you lose one pound of your tax free allowance. So in fact you're paying 60% tax, not 40 because you're paying it on more money than you realize. That was designed deliberately to kind of mislead the the public so that he didn't have to confront the idea of having a 60 pence top rate tax and mislead the newspapers a little bit. But it also inadvertently misleads the large language model. So you ask this question, it says top rate in the UK is 40%, 40% after the tax free allowance is this. Here's your wrong answer. So I've created this little bit of arithmetic that computes the personal allowance, how that diminishes and subtracts that. And it's only arithmetic, but it's already too complex for the LLM. And now if we go back to I've got my plugin enabled at the top, having deployed it, I can then say in here what is the tax I would pay in the UK on an income of. Let's make sure we're in that bound £112,000. And hopefully what it will do here is recognize my custom tool. But I that I created that is at the moment personal to My account and it will. ChatGPT is definitely being very slow today. Hopefully we'll get something happening. Otherwise I will jump to a pre created example. Really? Is it not going to do anything? This is the fun of doing things live normally. You get a response out of ChatGPT in a couple of seconds, but I guess it's quite busy at the moment. Okay, well while that's thinking let's I have fortunately here a a pre confused one actually I didn't name it the first time and it said use unknown plugin but in the hopefully in a moment on my browser it's going to say used whatever I called it UK Income Tax calculator. And behind the scenes if you view this you will see it calling with a parameter saying income goes to 112,000 and it'll get back some information that says personal allowance goes to this amount. Tax goes to this amount. Post tax income goes to this. Has it done it? We're starting to think now it's used it. So I can look behind the scenes here and see the conversation that's happened and you can see here that it's figured out which is the parameter to pull out of my conversation and it's got the computed value back. The text writing is being very slow, but I'm sure it will use those numbers within its essay. So there's a couple of things to observe here. One is, this is private knowledge. This particular thing is a well known bit of knowledge. But this could be any algorithm. This could be my secret algorithm for making some decision about whether I should buy a share or not. And that can be computed on my server behind an endpoint that hides how it's done. All the interface saw was the the request and the answer, and this was doing arithmetic. But once we're in the code world, the world of computational code, this could do anything. So this could access my private databases. It could access something actually in the physical world. It could drive a robot or it could read some telemetry from something. Once we're in that computational world, everything that we've taken for granted for years suddenly becomes available to the large language model as mediated by you deciding what the tool is capable and not capable of doing. I said it might be a very bad idea if you have if, if you allow arbitrary access through such an API. So if your robot could be controlled to do anything, then some, some poorly imagined piece of large language model control or some malicious user could make it do something bad. So we want these things typically to lock down what it can do in within with some safety. Another thing to observe is here that while this is more powerful with what's called prompt engineering, where I could give text that describes when to use this all, when not to use the tool, what it does, how to interpret the results in its minimal form, I haven't had to do any of that because already it's managed to infer from the name of the tool and the name of the endpoint and the name of the parameter. The LLM is already intelligent enough to have figured out the context in which you can use it, which is how it figured out. If we go back to this and we can see the answer we have now, that when I asked it this question, it knew enough to know that that made the tool appropriate and that it had to pull out the income part of that, not necessarily the UK part of that sentence to pass as a parameter. So there's a lot of kind of inference going on. But it also reminds you of those sort of first lessons in programming where you're computer science teacher said always name your function sensibly. That used to be for human benefit, but actually here it really does matter. It affects the performance of this tool. If I give this tool a sensible name because it actually is being used in the same way as the human. Programmers like to be able to read the function names to know when to use the right one. So does the LLM. So it becomes a more important part of your practice now. So that was custom knowledge in a chat GPT interface. But if we want to build our own interfaces to embed something in our application or to because we have some custom way that, that we want to access it from, maybe not even from a human, but from a, from a database or something, then we also need to have a code interface to the same kind of concept. And here's how we've been extended the extending the Wolfram language to achieve that. So the here I'm calling again one of these abstractions on LLMs, which is LLM synthesize and that is just write an essay and giving it the essay I want to write. But this would I guess come out of your user interface if you're making a tool for people to chat to. But I'm declaring the tools that are available. So this is the equivalent of that deployed plugin that I had in the in chatgpt. I'm saying in the tools list in this case there is one tool and it's called City Population Finder. And I've given a little bit of prompt engineering this Time to say what it's for. It's to get the populations of cities. I've got a parameter which says it's going to be called city, but it actually has to be a city. And then I'm going to call the Wolfram knowledge base to just fetch the property entity value of the population of the city. So when we get the answer back, all of the texted down to here was pure large language model, the LLM in charge. But at that point it's decided that it needs access to the information from the tool. And so it's called the tool with the appropriate parameter of what is it? This is writing about Chicago and it's got back the hour number for the population of Chicago and then it's continued with its essay in the LLM world until it got to here and it's called the tool tool again and said, can you give me the population of New York? And then it's used that number and it's written the rest of it in the LLM world. So it's exactly the same thing as plugins, but this is under programmatic control. And again, once you're in that code world, you can do all kinds of things. So here, for example, same kind of idea, but I'm just using a different context. I'm using this LLM tool idea of I'm using set of tools. In fact, I'm being a bit more specific here and saying I want to use GPT4 and the tool is get the name from an email address. So my aim here is to say write an email to johnmol co.uk for not attending an event. But we want to write that email so it looks nicer to the reader. And so we've got a tool that says if you need to look up the name of real person, here's how you do it. And then what my tool does behind the scenes is use Wolfram language as a bridge to SQL. And I'm just doing a select statement from a database that's on the target machine and you can see here that it does its job. Probably I need to give a bit more prompting to say who it should be writing from, but it's written this template that says dear John McLuhan and it's got my name here out of the database. So that was never provided in the prompt. It's come from the tool. Let's do one more, which is essentially the same thing again. Another example, just a different context. I'm making a little Q and A thing that understands postcodes. So My tool here is to call GPT4 with a postcode lookup tool that says find facts about a postcode and it needs a postcode which has to be a string and then it's going to call a resource function. I wrote some time ago that behind the scenes calls the UK Postcode Lookup API. There's a chain of events going on here that when I say what's the constituency of this is the Wolfram Research Europe offices Postcode where I'm sat right now that the LLM figures out that it needs tools. The LLMs innate knowledge recognizes what a postcode looks like. So it recognizes this string is probably the postcode. So it knows what postcodes in different countries look like and passes that to my tool. The tool uses downloads the function that I added to our function repository to bridge the gap to the API. It calls the API which I don't even know who owns the API, the UK Post Office or somebody like that. That looks up the response gives back in fact a whole bunch of information like the geolocation and, and, and the parish and the street address and things like that. And then it's used that to pick out only the bit of the information that the question demanded. What's the constituency? And that's one of the things you get back is which, which government voting constituency are we in? And we're in the Whitney region. And I've asked various other questions about Postcode, what parish it is which is basically the, the village, basically the parish is the area church. So this is the village that our offices are in. And then I've asked for the coordinates and it's got back Geolocation which is translated into English from our, from our kind of usual geo markup. But in the end this is all these are all doing the same thing tools. Can you have multiple tools that the LLM can switch to program and you can expose those programmatically so that you can put whatever interface or weave them into other code and make these LLM capabilities available with the LLM in charge or with code in charge. Now I said at the near the beginning that I thought one of the amazing and under appreciated aspects was handling unstructured data. And that I think is most of what I've been doing up to now has been either telling stories, doing text synthesis where it's trying to write content or it's been chat type thing. So they've all been kind of human language interactions in one form or another either on the input or the output or both. When we get into into more data science. One of the challenges for data science has always been that data is always in a mess that you wish it was nice and clean. But either the ontology is messy or it's written in in free form notes or you know, all of these things that need cleaning up. And LLM purely as a computational infrastructure helps to deal with that. So here's a kind of simple example of how to do that. I've got a new paradigm that I'm using which is in this case LLM example function. The idea of that is we're going to write a function that is powered by the LLM, but the way we're going to describe it is not using English instructions, but I'm going to give it examples of what it will see and as you can mix both. So I can do what I did before of giving instructions saying write a story like this, or I can give examples, or I can do both and say, I want you to extract the age and height and name of a person from the text. And here's some examples of what that should look like. In this case it works well enough that I've just done the example. So here's something you might see. John is a 52 year old male who is 6 foot tall. And what I want you to do with that is to make a wolf list with a Wolfram language association with key value pairs. Name goes to John, age goes to 52, and I want that 52 to be wrapped in quantity, comma, years. This is the Wolfram language syntax for units. And the height is going to also be wrapped in units. So again, an example of how to mark that up. I've now given it a little bit more partly to reinforce the example and partly to show it what to do in other situations. So here I've got an example where the age of this person isn't mentioned. So I'm showing it. In this case I want you to use age goes to missing. And here I've got third example which has two people in the same sentence. So I say, well, what I want to do is in this case see two associations, each of which have names. So group the information together. Name Mike, age 25. Name sue, age 23. And in this case both of them are missing on the height. And from those three examples it's enough now to be able to give it unseen data. Here is my slightly convoluted story that talks about several people and how they're related and has that information embedded. I've used this function, I've just created through three examples to generate a computable data set. All of the extraneous information stripped away, the facts brought out and they're in computable format. So I can immediately do things like what's the shortest person mentioned? And convert their height into miles. And because everything's marked up correctly with missings and quantities, that just works seamlessly without the messing up or the fact that actually these are in the same units, but they could easily have been in different units getting in the way. So this is as a way of handling unstructured freeform notes kind of information is incredibly powerful. You can now take that, all that knowledge that's locked up in essays and pull out these salient bits ready to do traditional computation and traditional data science on it. Now that paradigm of instructions and examples is really very rich. So you know, there I'm saying identify bits of information, pull them out. That's a pure kind of data extraction exercise. But you can use exactly the same paradigm to do all kinds of human like processing job. So let's, let's show both of these. So this was a proof of concept mini example I wrote working with a group called Euphonia who do speech recognition for medical chatbots. And their problem was that the off the shelf speech recognizers are generic. They don't, they aren't trained on medical conversations, they're trained on conversations. And so more obscure medical terms tend to get mistranscribed. And so they gave me this example and you can see what I'm doing here, it's exactly the same as before. I'm mixing some prompt engineering to describe what I want the thing to do that I want it to correct transcription errors. And then I've got two examples, one that's in the topic of hearts. You have a tastemaker and I'm telling it that actually that should have been transcribed as you have a pacemaker. And another topic of heart. And in this case I've got Asian populations might have been atrial fibrillation. And I think it's kind of amazing that from just two examples and a bit of description already the LLM sort of says, oh yeah, I get it, I know what you're on about here. And now when I give it topic of eyes, I have cats and a glad aroma, it's figured out how to map that to something more I related and it's mapped it to I have cats and I have cataracts and glaucoma just from those two examples. And of course if you're doing this properly, you might give it 10, 20 examples, maybe 50 examples to really kind of fill out its space and give the training. But just two is enough to achieve this. Another kind of error detection, again a mix of prompt engineering examples. This is a typical kind of just fact checking exercise. I've got some data here that's going to mention some diagnoses and drugs and we just want to pick out do any of these look inappropriate? Check whether the drugs mentioned are incorrect for the described symptom, return the drug mentioned and whether it's right or wrong. And I've given it some examples. The patient had a headache so I prescribed aspirin. I want to say the drug is aspirin and that was correct. He's taking warfarin for his blurred vision. I wanted to say warfarin is a drug and that's the wrong thing because warfarin is a blood thinner. It's used to avoid blood clotting. It's nothing to do with vision. And again from that example, it's enough already to infer what to do with this. The patient had a broken leg and I gave him some loperamide. It says the drug is loperamide, but it's the wrong drug. And here's an example, it's using it correctly. The patient was in need of an upset, with an upset stomach. I gave them leperamide to help. And that actually is the correct use of loperamide. So that kind of anomaly detection, error detection, you know, again you could imagine, I mean even, you know, there's an innate knowledge required here that this is using all of that training in the names of drugs. So even, well, the first one you can imagine, or the data extraction, you can imagine taking some 16 year old intern and saying here's what I want you to do. And they probably do a good job here. You kind of need somebody who's got some basic medical training and tell them what to do. And the LLM seems to come with enough basic knowledge to be able to achieve it. How are we doing on time? Let's show a couple of these. So categorization is another thing that you can do with this kind of methodology in a data science world. I've had these kind of problems in the past where you have two databases and they have different ontologies. One, I seem to have lots of medical examples here. One is a database of drug names that perhaps were prescribed. Another is a database or reporting you want is the drug type. And we need to join those up. And in the past that required building Big dictionaries where you had to say aspirin is an analgesic and we warfarin is a blood thinner. Now I can just describe that in words. I'm creating a function whose job is to convert the drug into a purpose of drug. A little bit of prompt engineering to make sure it does the right thing and not write essays for me. And now I can apply that function just within a piece of code. So I've got Wolfram language code that says make a pie chart of the frequencies of the purposes of this database of drugs. And while most of this is Wolfram language code, behind the scenes the LLM is doing that mapping to say what kind of a drug is aspirin? It's and analgesic. Warfarin and heparin, they're both anticoagulants. And I've done that data mapping entirely with that one line of English language description. Let's do one more. Same kind of thing of categorization. In this case I'm describing an offensiveness rater for say a social network post and we want to block things that are too offensive and make our social network nicer. And again, it's just an example function. I give it a bunch of examples of different scorings according to my personal opinion. So you're a nice person in zero offensiveness. Up to my worst one on my list is I know where you live and I'm coming to kill you. That's 10 out of 10 offensiveness. And now I can give it these unseen phrases and it tries to map what it's learned from this to say, well that looks like around level six, it's somewhere around the you're being compared to somebody, something offensive. Let's not do an example but I talked about quickly here. It does open up a lot of interesting ideas in edtech as well because now assessment becomes possible to parse very complicated things. So what is an omelette? That's a really open ended question. Does the person really understand? You can have a multiple choice very easily but to be able to allow somebody to say well it's a foods containing eggs or it's a meal that's made with eggs or a recipe for an omelet would be eggs, salt, cheese and ham. Those are all possible answers. Whereas saying it's a weapon of mass destruction is not a good answer. Being able to handle that fluency of possibilities in a question, give it a right or wrong answer is now made very easy through LLMs and we have that already hooked up within our question. Framework. Now, a slightly different angle that I've touched on already is that if we go turn back to chat again for a moment. Chat is mostly thought of as a kind of human to human type exercise. But actually, as you've already seen from our tool behind the scenes, you can synthesize code. And so you can now use chat as a way of talking to code, or talking or asking questions of code. And the plugin is one example of that. But I'll show you an environment that we've built to try and to make that useful. I'm going to start here a chat notebook, which is now part of our environment. So this is one application that we've done to make it easier for people to learn Wolf language. But that's just one application here. So here I'm going to say let's do something here, like plot the differences between the first thousand time numbers. Let's have a go at that. And what our Persona does here is its job is to help me to learn Wolfram language. So it's going to say, well, you could use the prime function, you could use differences. It's trying to point out the commands that I could use, but it's now synthesizing the Wolfram language codes to achieve that. And hopefully in a moment it's done that and it's going to make a plot of the result and we're going to get our answer concluded. So this is taking two or three lines of Wolfram language code, but I've been able to do that entirely in an English language way. And in a chat environment you want to be able to reuse context. So let's go on and say show the moving average with a window size of 200, make it green with grid lines, and it's going to take the context of the previous request, modify the code so I can look at the code. But at the moment it's just kind of quietly keeping that to itself. We're just going to kind of continue to modify this thing until we get what we want and hopefully we'll get an answer here. And while it's thinking, I'll just queue up the next question, which is show me the code. So once we've got it into a state that we want, Then hopefully in a moment we will get a picture. Well, it wasn't quite what I wanted. Let's just do one and wasting time here. But let's say show only the moving bridge, so I can kind of backtrack on what it's done a bit here. And let's get that to show me the code because I'm going to assume here in advance it's going to do the right thing this time and at some point I'm going to get the code that I wanted entirely through conversation by tweaking it. So I've said that's not quite right. I want this, I want that. And eventually I've now got the thing that I was aiming for here which is nice moving average of the differences of the primes and we can see that kind of. I think it's. It's. I can't remember if it scales as square with a log. Maybe I could ask it to fit a line to that. But. But here's the code that I needed to generate that and it's deciding to explain that code to me. I don't really care. I will click on copy down and now I've taken the code out and I've been able to use it. Now the Personas that is built in here is one choice. We have a couple of built in of a code writer which is a more expert level code assistant that goes straight to the code and doesn't bother with all this explanation. Something that perhaps I would use as I'm familiar with Wolf language. I don't need the explanation, but I am lazy in what code written for me if I can manage it. Plain chat, which has no Persona. But where this gets interesting is to be able to create conversational interfaces to your own code, to your own endpoints, to your own to combinations of the two with goals that are embedded into the notebook and and have access to private information. And we have a mechanism to wrap all of that up in a Persona. As a demo for this session, I've created this custom Persona called Notebook Reader. So this isn't one that ships with the project product. I should probably improve it because it's a bit basic at the moment. But I'm going to open a notebook that's already set up to use that Persona and I've got my prompt in here which is read injecting relianceability nb, which is the talk notebook that I've been using through this talk and summarize it in 30 words. So in this I've told it that its job is now to help me to read documents. It's not going to write code. This one is particularly about helping me to interact with notebooks and I've given it a special tool that can go and find that file within the working directory, read it, convert it into plain text and then in this case, it's given me the summary and now I can interrogate that and further by saying make a table of. Let's do something like how many slides are there? Slideshow notebook. So there were 12 slides in the notebook. What, what are their titles? And here's the main titles that I had on each of the slides of this document and hopefully the one we're on will come up at some point. There it is, conversational computation. The slide that I'm on now, I could ask other things like extract facts from it. I could say something like who is the author? Because I had that in one of the cells, it's been able to read that piece of information and pull it out. This ability to quickly extract information from collections of documents, that's just another Persona. If you start using your imagination and it's the combination of private knowledge, private databases, the prompt engineering to describe what you want, the interactivity of being able to have a session that understands the context of, of what you've said before. You'll notice here that once it's read the document, it doesn't have to keep rereading it to answer this. I've told it already that unless I ask for a different document that it just should use the information that extracted the first time for efficiency. All of that knowledge can be wrapped up into some kind of conversational environment. And in our world as well, these Personas can be switched. So I can change this. I could switch immediately into for example, a code writing assistant if this was relevant here and say, I don't know, I haven't tried this. Do I dare do a code assistant here? I'm going to try something here that is make a grid of the slide titles. Probably should have rehearsed this beforehand, but here we're using Switch Install Wolf language evaluator because I changed Persona here to being the wolf themselves. Hopefully it's going to pick up the context of taking this information and synthesize some kind of import statement. Or let's have a look and see. I have to let it finish. We'll see what it's done. Well, it's done something interesting. It's decided to pad it into using a text padding and I think I'd want to give it a follow up font. Say no, I want to use convert into a list of lists and use a proper grid function. But let's see what it actually synthesized here. Okay, this is kind of interesting. What's actually done here. It's taken the, it's broken the thing into a list of strings and it's partitioned them into threes with in steps of three. Well, it's done some weird partitioning and then put those into a grid. It's done something interesting. It's not quite right. Well, no, actually no, I'm going to take that back that the actual output that it had here before it tries to turn it into a text summary of the output was in fact the right thing and if I copy that down we can get the actual grid out. So taking live risks seems to have paid off on this occasion. So I think I'm at the end of the hour that I plan to speak at you before I go over to the Q and A session. So now's a great time to ask questions as I just wrap up and I will try and take as many of the questions as I can before we run out of our allocated time. As you can see, what we're trying to do is bring together the LLM kind of modern generative AI world together with the computational world in every different permutation of interface driven, code driven LLM driven, using the LLM to drive code, using code to drive the LLM, making that available as, as output reports or in chat environments, or as APIs and programmatic delivery and just find all of the important ways and abstract their joining up. The slightly less complicated way of putting it is to fall back on the kind of old metaphor of the human brain that the left brain is creative and the right brain is logical. And you know, in this world, the Wolfen world is the slightly aspergic logical right brain and GPT is the charming poetic creative left brain. And like any good rounded human being, we have some mixture of both and calling on both to solve our problems. And that's really the aim that we want to have. So thank you everyone for, for patiently listening to that. Who's still here? Let me now go over to the look at the questions and see if I can take some of these. Do you fear that OpenAI takes over the customer interaction? They might want to share from the pro subscription fee and can make it difficult for users to be aware of the monetization, etc. So. Well, first of all, as I've shown with the code access because we've built the paradigms and for programmatic access to the, in this case OpenAI APIs, you can set things, I mean here I'm doing it through code, but if I set up customer interface I can completely hold the customer interaction and we have a paradigm I didn't show which is for how to represent the context of a chat. So if you're writing your own chat environment that wants to be able to manage all of that context of previous conversations, we have mechanisms for dealing with that. So you can put the whole thing behind, as I did here, a notebook, or you could put it behind a cloud notebook or you could make some React JavaScript front end. That is the interaction. So that's no problem at all. Already behind the scenes. Of course, right now OpenAI is monetizing what I do because I have an API key for OpenAI that enables all of this. And for chat type things, it's incredibly cheap. My entire time of writing code and playing with examples, I doubt if I spent much more than a couple of dollars in the last couple of months. When you scale, if I want to go through an entire customer database of interactions and do fact extraction, that fraction of a penny per query suddenly can add up quite. If you want to go through a million records and it cost me 2 cents a time, then we're at thousands and thousands of dollars to do that. So one of the important things that I probably pointed out, but if I didn't I should point out again, is that one of the examples I did that our strategy here is that the. And that's not one of them, is that the. The model that we're calling is abstract is just a detail of the abstracted idea of using an LLM. So we want to really capitalize on the marketplace that is going to rapidly form over the coming year. So at the moment, as I said, at the moment we only support ChatGPT, but we already have two or three other connectors that are almost ready for release and in a couple of years this will be a non issue that you'll be able to go to whichever provider you want. And there are some interesting models that are coming out that can be run locally on your own hardware. Now there's a bit of a trade off because you might find that one model is better and for a challenging task is the only one capable of having the understanding and it's worth paying more for that. It might be that the thing that you have is relatively simple and you can get away with an open source, locally executed model and pay nothing for your execution. And there's also issues in that to consider about privacy. If you have compliance issues that stop you sharing data with somebody else, you can't use the public GPT4 because that's going to OpenAI servers, you might be able to use the Azure private instances of ChatGPT that we have already some working connector to that would be at least inside your Azure instances. Or you might have to go and use something like Falcon as a private model running on your own servers. But that may be penalties in terms of quality as you go through those choices. So I'm not concerned right now. As a short term thing, we are a little bit bound into OpenAI. Do I fear the possibility of vertical integration by OpenAI that they start to do symbolic AI themselves? Well, I would love that to happen with us as a provider, of course. And you know, we're were connected up with, with the people at OpenAI. I'm, I'm actually not in the teams in those conversations to know exactly what collaborations we're talking about, but it would certainly be my hope that, that in fact we could help to serve that need for them because we do have in the end the best symbolic AI application out there. And the other thing that makes me not fear it too much is that that in the end it's not just the computations that are tricky, but there is a kind of serious representational problem of being able to ask for the computation you want in a programmatically unambiguous way across many, many different possibilities. And we've largely solved that. The Wolfram language is a kind of towering achievement of computational representation, even aside from the algorithms that it then invokes that, you know, we've built this whole language of being able to represent all kinds of data quantities and so on. And while there are kind of open source free computer algebra tools and stats packages and visualization packages that they could wire up all individually, there isn't anything like Wolf language for bringing all of that together. So the chances of having the kind of ambiguous misfires and tools that come back and say, oh, I'm sorry, not a number because they called the R package for doing statistics, but in fact didn't notice that it was a symbolic value was amongst the data. And you know, all of that becomes a horrible mess if you haven't dealt with the layers of automation that we've spent 35 years building. So I'm fairly confident that our head start is enough to not be threatened by this. Although no doubt there will be steps in that direction because in some ways GPT4 is a collection of separate tools orchestrated behind the scenes that are branching and doing multiple different things at once. Okay, Peter, I will follow up with that. Say a comment that ChatGPT is having elevated errors in their fixing API. Yeah, there is one thing to realize in this world which is somewhat settling, is that this is still very much an emerging field, that this was a lab thing only, I don't know, maybe nine months ago or maybe as long as a year ago, this was essentially something that seems to be going on in labs and suddenly it's. In the last 12 months it's become mainstream. So there's still quite a lot changing. And I do see that too, that chatgpt sometimes changes APIs and we're kind of having to patch things and is one of the benefits in a way of having an abstraction layer is you don't have a thousand people having to do their own patching. We're trying to keep up with that and the abstraction also once we have these multiple backends will allow, if there is some, some breakage because things are changing too fast upstream, you can just switch to another provider until things return to normal. So those are all benefits of this idea of abstraction. But I agree it's a little frustration when things are bleeding edge. But. But I don't think that will last. How can I install the Chat GPT plugin on my computer? What you need to do is do the following commands. You do packlet install get what it's called find up here. Packlet Find. It is. It is the Wolfram ChatGPT Plugin Kit. So if you do package install Wolfram ChatGPT Plugin Kit, then that will fetch the plugin kit from our servers and install it into your copy of Mathematica. I would recommend being on version 13.3. I think it might actually work with version 13.2, but I'm not certain about that. It's unfortunately not very well documented. So download the notebook that's been provided here and you can copy my example, which is pretty much all you need to know to be able to. To be able to get it working. You do need to have an active Wolfram Cloud account, although a free cloud account is sufficient for it. You can also. I will show quickly here, quickly that there is another mode which was plugins, so it was this one. Instead of ChatGPT plugin cloud deployment, there is a ChatGPT plug and deploy and that will start up a local host server on your own machine. So you don't need to then put things into the public cloud. It does exactly the same thing, but it runs serving off your computer that you're sat at as you're working in the Wolfram language. Just by way of quick explanation, I will show you what you do to install that. Having run that, you get this URL that gets generated, you copy that URL to the clipboard, and then you go to ChatGPT, you do new chat, you do plugins, pluginstore, develop your own plugin, and you paste the domain into that box and there's a whole bunch of security warnings that you have to say yes, yes, yes to. And then that's all it needs to make it available. So it's really, really simple to install. Okay, so apparently we were supposed to end after an hour, so I've rather overrun my thought. I will go through the questions that are still open and if you have if you're registered so we have your email address, I'll try and send answers to the questions through email in follow up. Otherwise, feel free to write to me directly here and I'll happily take questions offline. Thank you very much for visiting, and do attend our other webinars in future.