Providing Computational Superpowers to Enterprise AI

Thanks everyone for joining us today, particularly those joining us live. I know there's lots of good talks at the Wolfram Tech Conference this year and I appreciate you all taking the time to join ours. My name is Zeb Smith. I'm a senior project director for the consulting arm of Wolfram Research and I'm here with my colleague Jesus Hernandez, who's a principal consultant. Just a quick blurb about us. I've been with Wolfram Research for going on 14 years. I started as a developer for WolframAlpha, which we'll touch on later in the talk and then I done for the last decade or so I've been with the consulting department and I'll talk more about what we do as part of this talk as well.

Jesus yes, hello, I'm Jesus Hernandez. I'm also part of the consulting group here. I've been at Wolfram for 12 years now. Starting the end of last month I was formally working in some of the Wolfram Technology group and then in a partnerships group and I've been with the consulting group for about the past year and a half.

Thanks Jesus. So I promise not to bog us down with too many slides. I think I have 14 total for our hour long talk today. I hope to do we hope to show lots of cool demonstrations as the main focus of what we're doing today and then hopefully we'll have some time for some Q and A as well. Our main talk today will be highlighting as there are many of the talks at the tech conference this year, the capabilities that are provided with the emergence of the powerful LLMs such as ChatGPT, Claude2, etc. Within the last year. Our consulting part of Wolfram Research has been growing steadily for well over a decade and we've been in the space of delivering data, science, applications, smart systems, AI assistance during that time and with the emergence of LLMs, we realized the potential that they provided with really enhancing the solutions that we were already providing. And what we're going to do talk about today is based on the technology that we have built in the last year. And by we I mean Wolfram Research, much of which is discussed in the various talks of the tech conference this year. We have now want to propose a platform really for providing LLM backed tools within enterprise environments. So in your organization, for your business, working on your data, all inside the privacy of your infrastructure, that's what we're going to talk today. We're going to focus on the cool stuff you can do with that technology in your stack. But we're Also going to discuss how we integrate with your technology stack, how we provide that technology. The story of Wolfram and large language models is, if you listen to Steven's keynote, it's pretty well described there. There's a number of blog posts that he actually did earlier this year as well that really describ our evolution of what we've worked on this year. Back in the spring and the March time frame, ChatGPT launched their support for plugins and Wolfram was one of the first plugins available for Chat GPT. And many of our demos today are going to be shown from that environment and in that blog post and since then, and many of talks today and some of the training, et cetera, that have been going on the tech conference, we've been, you know, you'll see examples of how powerful that integration can be when you have AN LLM like ChatGPT with access to a tool like the Wolfram capabilities to do computation, and the results that are provided from that are significant. We quickly had support for OpenAI. Anyone who could make a plugin, you can host it locally or you can host it somewhere on the web and you can have your plugin. And working with ChatGPT as well, we launched tools to do that for our customers. You can do that in a local environment or in our public cloud. And there's, you know, good support and discussions in the tech conference, as well as documentation for doing those sorts of things. And all while the time that was going on, we were in the process of building a suite of new functions in the Wolfram language to integrate with LLMs. This includes support for having an LLM use a tool much like you see in ChatGPT or in things like LangChain within the Wolfram technology stack, you can call out to an API, respond to it, give it the prompting, and respond to it when it decides to use tools and act on those tool calls, all within the Wolfram language stack. And then finally sort of a user interface for all of that and the Chatbook framework, the chat notebooks that were came out over the summer and I'm sure have been shown many times in the tech conference. A very powerful way from, in the Wolfram language to work with ChatGPT in a very similar and work with an LLM very similar to what you see online in ChatGPT. So based on the technology stack that we've developed and our experience working in these sorts of various use cases, we're going to talk about today, we eagerly develop various demos and, you know, we've had multiple conversations with governments, organizations, researchers over the past year who are interested in some of the use cases that we're going to talk about today. We're going to line out a few of those use cases, the first use case, kind of quickly and then we'll get into Jesus, doing actually some demos. And the first one is very similar to what you see in ChatGPT. Taking all the use cases you've seen for the TED conference that people have shared online with the power of having the wolfram plugin in ChatGPT. And then we take it beyond that. What other tools can we add to that and what does that bring to the table? The first thing, as is well known, LLMs can have a tendency to hallucinate. They can, they can make up information that's incorrect, they can try to do calculations, and they aren't really good at doing calculations, typically speaking for most of these. And, and so they need tools to provide accurate data and to provide that sort of calculation support. And when you give them those tools, essentially what you're doing is providing natural language access to programming and algorithm development, proprietary data, a built in suite of data analysis, machine learning capabilities available on the Wolfram language, as well as our extensive plotting, graphics and visualization capabilities. We can connect, we can create connectors to your proprietary algorithms and APIs that you might want to have the LLM have access to. And in some cases, as we have in our public plugin, it makes sense to have the knowledge base that comes along with WolframAlpha and what it really opens up and allows us to do. Your subject matters experts are allowed to have the ability to work with an AI assistant and work on complex problems in a back and forth fashion where the subject matter expert can see the intermediate steps that they're working through with the LLM, check the validity of those and correct any errors that we might be making as we go through the workflow, solving the problem. And in the end you can even have a workflow that you might want to save for repeated use or an algorithm that is created that you want to use in other ways and that allows you to have essentially not just a tool for a subject matter expert to ask a question or work on a problem, but actually a tool creator. Is what we potentially have here. One of our, you know, at a high level, one of our, sort of the easy ones that we got, we've gotten for a long time. You know, I have a CEO has a question and they, you know, to get the answer, they have to engage multiple departments in their organization, they have to retrieve the data, they have to give that data to an analyst. The analyst has to maybe come up with a model, and maybe that CEO gets an answer a month later. And we've had repeated queries and we've developed solutions to try to alleviate this burden. But now, with the presence of LLMs, as we'll show on our demo demos, we've really opened up significant capabilities beyond what we've been able to provide before. So I'll let Jesus dive in. We're going to show examples. He's going to start with Chat GPT and he's going to start with just the Wolfram plugin and they'll work some use cases that are probably interesting to, you know, business use cases in general, but showcase the power of what you. What happens when you have this integration in place.

All right, thank you, Zeb. So, just to provide a bit of context here, in case you haven't seen it before. So what I'm looking at here is within ChatGPT's interface on OpenAI, when you first go there, just to kind of show you what it quickly would look like, it would look something like this. If you're subscribed to ChatGPT plus, then you'll be able to access both of these and also access things called plugins. And what I'm going to show in this first demo is I'm going to use a plugin. I'm going to use the Wolfram plugin, which is freely available for anybody who has ChatGPT Plus. And you may have heard and seen talks about this before, but that's a plugin that provides access to both a Wolfram Alpha, but it also provides access to a Wolfram kernel to do computations in the cloud, so you can send it Wolfram language expressions to then be evaluated. So just to kind of quickly parse through, walk through, I mean, some of the stuff we have here. The very first thing I'm going to ask is I'm going to say to use Wolfram to import data from this URL. So I'm going to do is show that you can pull in external data and provide it to the LLM to then do computations with. You'll see here I prompted a little bit to say to use Wolfram just to make sure that it ended up using that plugin sometimes that not necessarily needed. And I also give it a little bit of prompting to say, don't show me the whole thing, just give me a brief summary of it. You could see here, if I open up this little box, this is going to show what code was sent to the Wolfram plugin and then what the result that came back was. You can see it actually made it knew how to form the correct import expression to then run it and also knew, oh, I know how to take parts of data. So I'm only going to show like the first three, for example. But it did say that this was data. So then we can kind of get here and it says what the first three rows of the data are. You know, has date, open, high, low, close, adjusted close and volume. And it kind of gives a little bit of background about it, what it could glean from when it saw up above. Now I'm going to start doing some basic computations. I'm going to take a look at it. So I say let's get the adjusted close columns from the imported data and I give it some hints to say re import things as you need. If it doesn't remember something, then I give it some clues of how I want this to look. I could peep in here again just to show you. It goes, it imports the data. It's then pulling out pieces of the data that I told it to, in particular the adjusted close. And then the response here. When you use the Wolfram plugin that's deployed in the public cloud, what it does is it returns the objects back as cloud objects. So that way it can embed it back in the ChatGPT interface here. And we can see here we get a dateless part of adjusted close series for Microsoft. So then what you could do is you could imagine being somebody who is, let's say a subject matter expert. Let's say you are comfortable in the field of computational finance, but maybe perhaps you are not an expert in Wolfram language programming, other kind of programming language. What we can do now is start walking through the thoughts that I have, the steps I'd want to do, the computations I want to perform and have the LLM itself kind of build up the code to be able to do it. So I'm going to start doing some things. Like the first thing I want to do is look at the log fractional. The fractional changes from each day. Now I take a log of it to get the log fractional changes and I want to plot a histogram just so I can see what it looks like. And again, I could peep in here at the code real quick. It does, it goes to that step of importing the data, does this. It takes the log and then it takes a histogram of those and it does all the things I asked it to do. So I could see here what it looks like. This looks pretty reasonable. So now what I'm going to do is I'm going to define a distribution based on these. Based on these fractional changes. And what I'm going to do is take that distribution and pull out random numbers from that distribution to perform kind of a random walk. I'm going to do 63 random values to go about a year and then I'm going to accumulate those changes and exponentiate it to then project like, simulate like a forward, a price pass. This is going to do one possible path that could be done. Again, I described this in fairly a lot of natural language English a couple of times. I'm going to highlight some Wolfram language functions that I know of to help it along the way. But for the most part, I'm using a fairly natural language to do this. And again, it built up all the code. I'm not going to peep in every single one of these because you're going to see this again in a bit. But you can kind of see it's still. It's slowly building up the code that I need to do in order to do all this. So I get here as a possible future path for this stock. And that's nice. That's a nice operation I want to do. Now, what I want to ask is go ahead and give me the Wolfram language code you did to do this so I can get a block of code. And what I'm doing now is as I'm building up this kind of more complex idea, this complex thought process that I have with these computations, I'm also now having a trail of code that I'm leaving behind. We see here this, with this copy code button, I can in fact copy this bit of code and then paste that in a notebook and evaluate it. And it would go through those steps like we just saw above. I could check to make sure that things are going like I expect them to do. What's nice is the LLM gives me a summary of what all the various steps were with this. In a sense, what we have is a path, well, to natural language programming is really here, especially if you are a subject matter expert and you know the computational steps you need to do. Being able to translate into Wolfram language is something that the LLM and in particular chatgpt4 here is actually quite good to do. So what we're going to do now is just make things a Bit more complicated, and I'll accelerate the pace here a little bit, is to generate now 100 forward paths instead of just one. And I kind of tell it how it needs to do that. I explain the computations that need to be done before doing all of that. And then what I want to do to sort it by, you know, various percentiles so then I can get the very quantiles of it. And then we'll show those in a plot. And again I get all these here. So now we get a list plot of the 5th, 25th, 50th, 75th and 95th quantiles, the possible forward paths for this particular stock based on the kernel distribution I had before based on log fractional changes. Ok, we can keep pushing this forward. Now what we can do is using these samples, we can start valuing, let's say, call options. And so we're going to do is we're going to define for it what a. You know, what we mean by this. So we talk about strike prices, and I tell it the kind of steps I want it to go through. And I give it some various operations. Here again, we're getting fairly complicated in the terms of steps we have. I have two paragraphs worth here of text to do. Then again, when I open this up, you can see it starts doing all those various computational steps that I needed to do. And it writes out here. Here's the various strike prices and steps of five. As it goes along, I can say, okay, I want to see a plot of that. It'll go ahead and give me a plot. Now I say, okay, you've done all of this rather than rewriting it all out. Again, I say, okay, we're going to do the same thing, except now we're going to call them calls. And there's a slight change in the definition for how you're going to do that. But it's smart enough to figure this out and to then go ahead and perform the computational steps to do that. So I have now a plot of calls versus puts. Now I want to consolidate what I've done here. So what I want to do is I want one function that does essentially everything we did up above. It's going to give us the quantile, it's going to give us the paths, it's going to go ahead and compute these calls and puts and put them all into a table for us. And so that's basically what it's done here. It's put it all into one various block. And then what I say, okay, I've done all of this. I want a single Wolfram language function now to do all of this. So what it does now, it generates this block of code with this simulation function as it's decided to name it, with all this sort of information in there. And it has the code in there to what it does. And it tells me what this code is going to do. It tells me all the various steps on it. Now I could say, okay, you've done this code, now give me a function that does the same thing. But now I don't have to predefine what the ticker symbol is, what the starting date and ending dates are. You'll see this occasionally. Sometimes it'll proceed forward and it'll try to make a function, but it'll make a mistake. And in this case, it realized it needs to adjust things to UNIX timestamps so that it goes ahead and does that. So now it's built the simulation function that now takes arguments and I ask it, okay, do that for Nvidia over the same time period. Okay, it notices, it makes a mistake, it gets a response back, it doesn't understand, it adjusts, make a different, you know, correct its code. And now we have the same thing for Nvidia. So it tells it all the things we have to done here. So I could go back up if I then wanted to share this with somebody else in my organization or then turn this into say an API function or something like that. I just have to copy, paste, press the copy code button, put this in a notebook and then save this. And now I have a function that performed all of this in an order of about 20 to 30 minutes. Somebody who knows the steps they want to do can have a back and forth with the LLM and walk through and then go do all of these steps, lead a trail of code behind which then can be used to perform further work and to share with co workers to do other sorts of things. That's a brief demo showing all this stuff that you could do using the basic, the Wolfram plugin and using some public data that I pulled from Yahoo Finance. In this case, the next example I'm going to show is slightly more complicated. Now in this case, what we're going to do is not pull something from a publicly available website, but let's say, let's connect it to a database here. In this case, this is a database under our control. We have in this case happens to be an AWS somewhere, but this is a database that has some example data from about 1,000 drugstores and so what it contains are daily sales and then information about those stores. So what I'm going to do is I'm going to start and ask it, I'm going to prep it to give it some context to what to do. I'm telling you what it is and what it does. And it says, just by what I've said here, it says, okay, what I need to do then is I need to go into the database so I can understand what the schema was, what the structure of that database is. I didn't tell it. This database has specifically these tables. I didn't give has these columns within the tables and all these sort of things. What it's going to do is go discover those sorts of things as part of this query database plugin, which is one of the ones that we created to support this example. What it does, it has functions that goes and explores things. There's an endpoint in there called get table names. It doesn't take an argument. What it does goes gets the table names that are in the database. It realizes, okay, some of those maybe aren't important. The ones I need are probably stores and sales. Then it goes, it says, oh, I'm going to go get the information about it. So it writes some basic SQL code to go get information from the database and then it gets this result back. So it now understands, okay, this is what is in the stores database. And then it can also get what's in the sales database. And it gives me a few rows here to kind of give me an overview of what the data looks like. And it describes what various columns might be. It does the same thing with the stores table. Then it notices, okay, there are four unique store types, A, B, C and D. And then it says, okay, what do we want to do? So I start go doing some exploratory data analysis. Again, I didn't have to understand or go through every single column of the database and describe it. I was able to walk through and get it. So now I can ask what were the total sales for each store type for the last available month of data. The first thing that it realizes it needs to figure out what does it mean by the last available month of data. So it does makes a request to do that. Then it goes does another step once it gets that result to then send another query to the database to do a select here where it's doing things, has to join various tables across. So it's a slightly complicated SQL expression that it needs to do there. And it gives me my answers here. And ask what else do I want to do? So I could start getting more complicated. What I'm going to do here is look through to see if I can notice any trends that are interesting. So I ask it to get me a dateless plot of average sales per day divided by the average customer visits per day, which is a very somewhat complicated expression to talk about. Then I wanted to normalize this data. One of the things that's nice to see in this particular instance here is what it's going to do is it's going to pull the data. It notices, oh, I'm dividing by zeros, because there are probably days, especially say, if a store is closed, when the number of customers it's going to be zero. It understands that it wants to correct for it. It's going to write better code to kind of take out that zero use case. So it's going to use query database. It's going to get the data, it's going to temporarily store it. In this case, it's going to use a local object because this particular plugin is also going to hand things off to a Wolfram language evaluator that in this case happens to be residing on my local machine. This is essentially a clone of the one that's in the Wolfram public that's available in the public cloud that I showed in the previous one. But this one is an instance that I can run locally on my machine. We could also do the same thing, having a version of that and let's say your private cloud as well. But in this case, what it's going to do is it's going to take that big chunk of data and it's going to save it to a local object, so then we can refer to it later. And this ends up being pretty important because now rather than trying to pass an entire chunk of data through these context windows, which can be pretty small, I can now pass it by reference, essentially. So you could see here, like what I pulled here was 731 rows of data that would be far too much to try to pass through this context window. But now all it really has to do is understand that it has this local object named essentially average sales and customers data. So it's going to do is it pulled the data and then my Wolfram evaluator here gets some code which says, okay, go look and get this local symbol and then perform these computations with it, and it's going to return an answer back as a cloud object before. So we could kind of see it in the picture here. And then kind of picking up the pace here again a bit. You could see that it's going to go through. We notice a trend here. And this is some exploratory data analysis. Like what? Why do we see these two distinct lines here? Perhaps it's due to weekends. So we normalize per week, average it per week. We notice that trend is still there. We look to see what else. Okay, there's this promo flag. Let's see if the promo flag has something to do with it. So what it's going to do is going to pull it now. It's going to pull the promo code column as well. And we look and we see, okay, oh, that's perhaps why it's been split. So we see that with promo, without promo, we get these two distinct lines happening here. Again, ask it for some more computations here. Let's get the total numbers of. Get a table of total customers, total sales, sales per customer. Let's focus on store type D and do some other kind of things. So in this case, I'm going to make a list plot of daily sales versus customer visits for store type D with promo, without promo. I kind of noticed there's a trend there. So let's say I'm doing some big basic exploratory analysis to see. One thing I could do is ask it to create a linear model. Again, I'm not telling it to use, you know, linear model fit or anything like this. It's going to know the Wolfram language functions to do these sort of things. I give it a little reminder about the kind of data that's going to be expecting. And then I could peep in here real quick just so you could see. It does. It knows it's going to use linear model fit, gets a result out. And I can get, you know, a basic linear model for this. I could ask what's the parameter table for this model? So you can kind of get estimates and P values and things like this. I can ask for the R squared for the model. In this case, it's probably really dependent. Obviously it makes sense. The more customer visits you have, the more, the higher sales are going to be. And then I say compare it to a model that doesn't include promo status and compute the BIC for each one. It says, okay. It notices that when you do all this stuff, using promo status as an additional thing to model against is obviously better to do. And I could go through, and I'm just going to quickly walk through the rest of this, do other things like look for seasonality and so on and so forth within all this sort of thing and just kind of keep repeating your logic there. So in this next example, what we're going to do here is build upon the kind of complexity of the code we've been doing. Now we're going to look at some demo data. This is a database that has used car sales in it. So what we're going to do here is now show how we before we did some really kind of basic computations there, showing plots and things. Let's say I have a specific task here if I want to do some machine learning work here. So what this plugin does is it provides some small guide rails to the LLM to effectively use the family of classify and predict functions within Wolfram language and then allows us to easily access some of the diagnostics information in those functions as well. So what we're going to do is look through, I'm going to, it's going to examine the data set as it did before to look through the schema to find out what it has. It finds there's a used cars, it's going to look for that. It finds all these various columns in there. And then I look at a histogram of prices. Here's a histogram of used prices. Ask me what I want to do for more. So I say here, let's make a model to predict used car prices. It's going to download all the rows from the table and I want to use 70% of this data for a training set. And I'm going to say I want this model so if I refer to it later to be used car 01. So what it's going to do in this first box here is it goes and does the database lookup. It's going to get all of that information. You can see now it's getting about 15,000, 16,000 rows. Now it's going to use the predict endpoint in this plugin. It's going to look price is going to be the feature, what product, what, what's the fractions used for training and what's the model name? It made a mistake here and when we look here, it actually formed this local symbol wrong. So it tries it again and now it's actually able to be successful this time and we get some diagnostics back out of it and then it returns back. What does it do? Like how many training examples, how many test examples, number of features, the training time, all these other sort of, you know, really good information to let you know as you're performing Your machine learning tasks, how it's going along. So one thing you'd naturally want to look at is a comparison plot of actual versus predicted prices. We can kind of see here. Now, I wanted to do some other things. Let's say compute the BIC score for this model. I prep it a little bit to know like this is what the model mathematically what I want it to be and knows how to do that. It goes through computes thing and computes a BIC score for this particular model. I guess I apologize. Let me go back up and say one thing that was important here is that it did specify what method it used. In this case it decided to use a random forest method. So that's gonna stay what I do before. So next I want to do is try to use a gradient boosted tree. I'm gonna specify what method I want it to do. So it goes through the same steps, walks through. You get a plot out of this, you can compute the BIC scores. I give it some prompting there, you know, it could compare, say which model might have a lower BIC score. Again, I'm going through this little quickly. And then we could see like a feature impact plot, basically see which features seem to be the most impactful when making these models. But a really neat thing I can eventually do is I have a model that I like. It's done the diagnostics I want. What I could do is then easily within a little line here because this plugin allows us to do it, is then deploy it to the cloud. So I have this API now that contains a Wolfram language predictor function. I could go onto the cloud and I've pre put this in here so I don't have to fumble with copy and pasting. This is that API. I could submit a entirely new car in there, you know, send that to the Wolfram cloud and I get a price back out of it. And again, I did that all from, you know, within this chat GPT interface and then finally of these kind of chatGPT ones that I want to show. It's interesting to talk about. We've talked about connecting to various, you know, groups of functionality, talking to proprietary data. In particular, we could talk here about connecting to proprietary functions and packages in your. In your company may have. So let's say you have your own custom Wolfram language function. In this case it's called my custom code. I'm not super creative about naming these sort of things. In this case, it happens to have some computational finance functions that are built into a package that's going to do some Fama French modeling inside of, but that the details that aren't necessarily important. But what it does is what I'm going to do is I'm going to do what's called few shot learning. I'm going to have my plugin is going to load the function, it's going to load the package, I should say it's going to read all the publicly available symbols, read their usage messages and use that to understand how to use these functions. Because this LLM has never seen these functions before. It's never been trained on these. These aren't Wolfram language functions that exist for anybody except for me at this point in time. But again, it understands, it gets these various functions, knows how to use them. And then I could start straightaway asking it to do various computations. In this case for Coca Cola, it's going to do a Fama French three factor model. I get these various parameters out of it. I could make various scatter plots. Again, these are all functions that are now provided within that package and ask for various diagnostic information. Again, these are things that were bundled up in the package that's going to be essentially talking about doing all of these kind of LLM examples within the chat GPT interface. This chat interface you see here. It is important to note though, based on the hard work that's been done over the past two or three months, actually probably goes back further than that. You aren't limited to having to do this within the ChatGPT interface. What you can also do is do all of this stuff that I showed before within Chat notebook interface. Here what this is. This is a version of that drugstore demonstration I showed before within that chat window. But instead of going through the ChatGPT interface to do this, I now have native access to all this LLMs within the Wolfram ecosystem and it also provides native access to the Wolfram language tools. In this case, I'm using custom tools like that SQL tool to drive these sort of things. You can see the names right up here being done. But if I scroll through this, you can see in this chat notebook do all the. All the exact same computations I was able to do before. But now I have this complete within the Wolfram ecosystem. And these chat notebooks could be deployed to private clouds where you could also do the same thing within a private cloud to have the entire workflow here from the chat interface system to the data analysis to the databases themselves being existed within an infrastructure that you would control yourself. And with that I'm going to switch this back to Zeb.

Thanks, Jesus, I appreciate it. I think there was some feedback that I was echoing a little bit. Hopefully it's better. Not sure what was going on. Actually. I'm gonna do another. I'm going to show another save chat, following up on Jesus's. Jesus's examples. But in this example, I'm actually going to show a couple here. In this example, the what we're doing not retrieving data from. Structured data like we saw from the Yahoo Finance example, but we want to retrieve data out of documents or presentations, things of that nature. And in this example we're piggybacking on some support that OpenAI has provided in ChatGPT. They have the concept of retrieval plugin, which is essentially a plug into a vector database where you have upserted your documents that you want to do retrieval on. And the first demo I'm going to show here is largely to get around some deficiencies of ChatGPT's website. But I wanted to show what happens when it actually makes the call. So we've asked what we're. We've given it here a vector database that has some news articles online, and we sort of have an idea of what some of the topics are that are covered in those news articles. And we ask a question about that. And from that we're getting what happens is we send out the query to the vector database, the plugin functionality, and we get out the top three hits and the text where the results for the question that was asked might be answered. And then the LLM reads these three snippets and decides from the data that's provided from the plugin what the correct answer is and provides those and the results here. And so we, you know, we double check this. It did do it correctly and it also links its source. So another important point here is obtaining the source. So we get the information in two places here as well as just the full document itself. So this is just some online documents. We have another example here. In this case we have some 10Q filings for some companies in the Dow, and we're asking questions about a corpus of these documents. So in this case, we're asking about a summary of Verizon's management discussions from our operations in the latest quarter. And you know, we have some. We're not just retrieving data, but we're using ChatGPT's capability to say, hey, actually I want you to be a little more concise. I'd like to have, you know, some very specific bullets here and we can get it to refine those and go through and pull out actually individual data items from the corpus of documents. And so this is another feature just very similar to the plugins that Jesus was showing. We're not just limited to databases and things like that, or we have more capabilities to do soft data using various capabilities. Let me go ahead and switch back to my slides real quick. And so what I'm showing here is sort of a workflow for the back end setting up of that plugin that we just saw. And we have a lot of experience doing this, as in our consulting group, processing documents in this fashion. We create a, you know, a control layer that's controlled by the Wolfram language on the outside and a workflow to process. In this case. Those tinqueue forms are HTML documents on the web. We convert those to PDF and then to text files, and then we chunk that appropriately, appropriately and upset that into the vector database. And that workflow is all controlled by the Wolfram language. And then you have a tool that can be called, just like it can be called in ChatGPT. And as Jesus was showing, all these tools that we have can be all integrated within the Wolfram language. So you don't need a separate stack to do that integration with your LLM. They can all be integrated in the Wolfram language. So I want to talk and move to a couple points about data curation. One of the big things that we do at Wolfram is data curation. WolframAlpha itself is, has a knowledge base that's curated by experts across a wide breadth of human knowledge. And we are constantly adding new data and keeping that data updated and have sophisticated ontologies to understand and be able to access that data using, you know, natural language. We not only do it for WolframAlpha, we've been doing that for external folks for well over a decade. Government institutions, private businesses, researchers, helping them provide ontologies and develop ways for accessing that data. Certainly in some of the examples that Jesus shown, you might notice that some of the column names were fairly well described. It made a lot of sense, pretty obvious as to what things were. We weren't dealing with too many column names, certainly to develop and use some of that technology on a very large database, maybe with nebulous column names, with tables that are just there to support linkage between tables, those things could and benefit from additional metadata. We have a lot of experience creating that metadata for AI systems such as Wolfram Alpha in the past, and certainly we're having a lot of success doing that. For LLMs now. And we aren't just using curation to help us work with LLMs, but we're actually using LLMs to do curation. So both for WolframAlpha, as Stephen mentioned, you know, LLMs are very powerful at this. Another powerful way is to actually retrieve data from soft documents. So similar to the chat experience, I was just showing that functionality can apply to a situation maybe where you have some government requirements to do some reporting, and the data you need to report is scattered across companies that have to do special filings, and you have to go into those companies and you have enter those reports and you have to find the data. And you know, up until even, even recently, even even now, we still have humans very much in the loop ultimately doing a lot of that work and finding that information. And we have a lot of people come to us and say, what can you offer to alleviate some of the effort that we're putting in here? How can we make this easier? And now we actually have some pretty decent solutions for doing such things. Here's a workflow where we have some definitive sorts of questions we want to ask, and we feed those into our retrieval plugin. We get these, the results from the documents, whatever corpus of documents we're shirking, tune for the answers for, and we retrieve those top K results and we provide all of that with special prompting to an LLM to generate the results, the answers. So in this case, we might be looking, for example, the revenue. In this case, we might not be just doing Chevron, but Maybe I've got 30 companies that I need to have this information for. So I've got to go through a corpus of 30. Maybe there's multiple documents per company too. So we're looking through quite a bit of information. Let me switch to another demo here. So in this case, you know, the idea is, and what I'm showing here is sort of the bulk processing. So I have, you know, I'm responsible for 30 companies and I have these documents that have the information I need. I have a workflow set up to process these every year and get the information I need. So let's say it's revenue for each of the companies that I need. I go in and do that same workflow I just showed where we ask it to get the revenue. That's what we're looking for. And ultimately what we're looking for is the answer. So this is the ultimate value we're looking for. And along the way we're generating where and the actual documents that we're Receiving this so that we can, so that someone can review this information once it's been produced and then go through it and, you know, verify and vet that the information is correct as well. So you probably still have a human loop here just to bet that the, you know, the LLM did exactly what you expected to and the information's right. But all the legwork essentially has been done for you here and it's significant speed up for people who are finding themselves in this situation. Another workflow that we encounter a lot is, you know, having a large document like that and wanting to summarize it, want to give a summary, and you want to be able to have summaries for all the companies so that you can see at a high level what the output of those. This year's report is. In this case, we're going to show some slides. So we've already done in pre process, essentially we've asked it to summarize some facts about those 10Q reports. And we said produce slides for your summary. And so we can go in and click on the Microsoft button and I'll actually have to switch my screen here and we get a slideshow essentially, and this is a slideshow in the Wolfram language. But the output can really be anything. The point is that we are automatically generating summaries from these reports that someone could review and use as needed. Could potentially even feed this into LLMs for other functionalities. So that's another feature and there's one more that I'll walk us through. So let me switch back to my slides for a moment. This, this one just shows the sort of. It's a very similar workflow. It's just the same workflow where we're, you know, automating some slide creation. But the idea is to summarize some information, found those reports. And finally we have one more demo that in this case we've used the tool a lot to interact with the user and find answers and such. But, but in this case we're putting the LLM sort of the end of the workflow. So let me switch my screen one more time here. And what we have here is a webpage and I've already evaluated in this case it can take a couple seconds to do so. So for the sake of, the sake of the time, I've already evaluated it. You put a ticker symbol in and it produces a report under the scenes. What's happening though is that we are pulling out a lot of Wolfram language data for, in this case, for the Microsoft Financial Information and then we've given it. We have an algorithm in place to analyze that data and identify some qualities about it. You know, how is the stock doing, essentially? And in doing so, we create a prompt for the LLM that contains the data that we pulled from our own resources, as well as a tone to set. So use an optimistic tone in the summary. And we're setting that optimistic tone about the algorithm that says, hey, based on the data that you're seeing here, should be optimistic, should be neutral, etc. And the result is to sort of write a little report. And in this report we have. Part of the report is to always create a specific graphic. And then we have the summary coming from the LLM, we can wrap that all up in a report. So we have some automatic report generation, which is another use case that we often hear about. So I think that largely concludes most of my our demos. I want to talk a little bit about how we provide this, how it all comes together and we provide this for your organizations. The main piece of the puzzle is our Enterprise Private Cloud, the Wolfram Enterprise Private Cloud, which we call the epc, that is designed to be hosted in your infrastructure behind your firewalls. And it acts as. It can act as the integration platform and it can be hosted either on your own data centers or in corporate cloud environments as Azure, AWS, Google Cloud Platform, etc. There's a question in every situation, you know, how do we integrate with LLM? Now we have clients that come with us that already have fairly robust chat interfaces, have integration platforms for LLMs. And if you're in that boat, you can certainly just call out to our tools with APIs, much as the you see in chat happening in ChatGPT. Alternatively, we the Wolfram technology stack is, as Jesus can show, can combine tools with LLMs and within our own stack. So that allows you to have a sort of contained solution basically in your own environment. Here's a little diagram that shows that the one thing I haven't touched on a whole lot, and I have a slide later, Wolfram doesn't actually provide the LLM. And if you have data concerns, and I saw there's a question about this, more and more LLMs are being offered for private use either in your corporate cloud environments, such as AWS, Microsoft, et cetera, which Microsoft has GPT4 models and you can even fine tune those and have saved models in your cloud service. Alternatively, there are also becoming more and more ways to deploy models inside your own infrastructure. Certainly that will evolve, but Wolfram is implied in the LLM we're integrating with it and we assume that in most these use cases, unless you don't care about sharing data, that the LLM will be within your, within your environment, in your, in your purview and not publicly available. Although I will mention that we have. There are use cases where certainly not always does the data have to be returned to the LLM. And it's not always the case that there are certainly situations where you can, you can do things where the LLM doesn't need to necessarily have access to the data. But lots of the workflows here, the back and forth that we've shown with the LLM, certainly the retrieval plugin, using it for data curation does send data to the LLM. The main Wolfram components, the Wolfram Enterprise Private Cloud is the main piece from us. You license that from us and we can help you install all that wherever you end up disallowing it within your organization. It hosts the tools for the Wolfram language computation. Any tools that we can use to hard or soft data connections, connections prior to APIs. Algorithms, they don't have to be Wolfram language algorithms is what Jesus. They can be, you know, written in any language, any stored procedures for data curation, etc. And it can act as that integration layer. And then the other big component is our Wolfram consulting team. We can help you actually build out, install and build out all the tools and that includes integrating them with the LLMs, doing all the prompting, etc. Our side of the company is time and materials. So any sort of custom plugins that we build for your proprietary data or that use your proprietary algorithms, all that stuff is owned by you. And that's the other big role that we have. Certainly an experience, people experience with the Wolfram language might not need as much consulting to set up their solution, but certainly that is available and we have a lot of experience and we've been working in this significantly in this space for, like I said, well over a decade. But with LLMs in particular in the last year and as I mentioned, we don't provide the LLM in our experience, GPT4.3.5 Cloud2 kind of have the best results for using tools and writing Wolfram language code in particular. Certainly models are advancing quickly and we, not just in the consulting group as the company as a whole, are investing quite a bit of effort and staying on top of the latest models, the latest, the latest trends in the technology space. So it's not just a one and done solution. You know, we'll keep our any enterprise private cloud that you would license for any solutions. We'll continue to get updated with our technology stacking capabilities as part of the typical maintenance of there. I have a bunch of links for additional information to reach out to. First off, if you are at all interested in anything that we have shown today and would like to learn more about it or potentially are interested in that solution for your organization, feel free to reach out to us on the Wolfram Consulting website. Fairly new website we have up. We've been doing it for a long time, but we've got a new website. There's a form at the bottom of the page you can fill out and one of our sales representatives will be happily to contact you. We give, you know, a handful or two demos a week really and have for the last year on these workflows and we've been doing it for a long time but it definitely has picked up given the plugin and exposure and showing some of our capabilities there. Certainly the main site we have a page that references a bunch of tools for AIs as well as the, you know, some other stuff for Wolfram Alpha, Wolfram Cloud, etc. I did put the slides and the handouts and I think I got confirmation that I successfully shared the handout this time. Let's see we can do we have time for a couple more questions and answers, so let's maybe dig into some here that have not been answered. Maybe. So there's a question about what prompts are public. So all of our prompts that you see with the Open AI chatgpt are available. So let me share my I'll share a difference. One second. Let me pull this up real quick. So for the OpenAI integration you have to post your prompts for the plugins publicly available. So if you go to wolframalpha.com wellknown AI plugin JSON, you can see the prompt that controls the the support for the open AI ChatGPT plugins. Some of the demonstrations that Jesus was using have access to Wolfram language that have more capabilities than what's available in OpenAI's ChatGPT. We created the API that evaluates Wolfram language code specifically for the OpenAI integration, and we've been continuing to iterate on that. That's actually the Wolfram language portion is actually originally architected by the with the help of our consultants. So we actually helped build Wolfram language piece and of course it uses WolframAlpha Endpoint 2, which is specifically designed for working with LLMs we have an LLM API endpoint and both of those continue to see development. There was a question about how to keep up with the prompts for the the integration as the model changes. I'll say that over the last probably three months it hasn't required too much prompt editing from us. It did take a lot of work to get the prompt to the point it's at here. But now that we have gotten here, we haven't noticed any significant changes with respect to the OpenAI model. And certainly while we have demonstrated most of our stuff today with OpenAI models, we have played with the other models, as I mentioned, and Claude 2, for example, works fairly well. The prompt you need is a little bit different in some cases. The plugins that Jesus was showing had some more capabilities and their promptings a little bit different. And certainly if you have your own model that you trained or you have, it might require some, some additional prompting. And that is something that, you know, we have a lot of experience with and consist with, you know, as part of the developing the solution for you talked about release of data. I gave a link to the Enterprise Private Cloud and the in the slides and you can go there to learn a little bit more about how that's deployed in your infrastructure. It's sized based on how much you use, so the infrastructure required, you know, like how, like the machines, how many cores they have, etc. Kind of depends on how much usage you have. And we have a lot of experience sizing EPCs for your need. We do have scalable solutions now as well. So, you know, those are things that when, if you're interested and we get serious about providing a solution or you're interested in EPC that our sales representatives and engineers can help you with for sure. I think that covers most of. Jesus has been busy answering stuff behind the scenes, so I think we've. We cover most of the questions that we had here today. We are very excited to show this. Like I said, we've been as consultants delivering solutions in this space for literally I've been working on it for a decade now and it's really been fun the last year to see how much more we can do with LLMs now, how much, how much further we've gone from where we were, you know, even just a couple years ago, you know, pre chatgpt. Right. So it's been an exciting year and it's been a lot of fun starting to work on some of these cool projects. So we'd be very excited to work on more if anyone here that's listening to this presentation is interested. Just reach out and contact us. And we'd be happy to work with you. Thanks a lot. I think that'll wrap us up then. Have a good day, everybody.