Integrating LLM into Wolfram Workflows: Process and Analyze Sustainable Development Data Efficiently

Hello and welcome. My name is Ben Kikert and today I'm with my colleague Maureen Bayer and our presentation is on integrating large language models into Wolfram workflows. Just a few introductory notes. Maureen and I work with two different initiatives known as Starting Up Good and SDG Counting. These are side projects that we hope bring about some some good in the world. The first works with supporting startups and entrepreneurs around promoting societal good in their business plans. The second, SDG Counting, which is what we'll look at more today, is about advocating for sustainable development data, particularly access, quality and usage. I will say that our presentation today builds a lot on the previous work that we've done and items that we've presented at previous Wolfram Technology conferences. We've done work related to how to interact with both structured and unstructured data using machine learning for data analysis, incorporating non traditional data sources such as YouTube and last year we talked about connecting to tools like GPT as part of your workflows. Hopefully many of you were able to view Maureen's presentation earlier using the Wolfram Stack and LLM tech to combat misinformation on the SDGs and ESGs. If you haven't, I encourage you to do that because it provides a great high level overview of how our team works and in practice, what some of these tools look like when they're deployed. The flow of today's presentation is we want to start with some concrete examples, give you some example code, and then from there move into more of a survey of the types of tools that our team uses internally to work with large quantities of data. Additional information and examples will also be included in this notebook. We have a lot to cover in a relatively short amount of time and so even if it's not mentioned verbally, I want you to know that there's some code in there that you might want to look at. And finally, just a caveat, given the rapid developments that we've seen not only in AI space, but also within the Wolfram environment as a whole, I must say that things are subject to change. There are parts of this presentation that I developed less than a month ago that are no longer relevant, and I assume that in a month or two we'll be seeing that that same change. In terms of what's included in this presentation, specifically we want to talk about how Wolfram tools can serve as the backbone for connecting with outside external LLM resources. Particularly, we want to look at Wolfram's ability to do pre processing, batching of multiple calls and then structuring information both as it comes in and goes out. From a content point of view. We're going to be looking at the voluntary national review process for the SDGs. These are documents that are submitted every year to the UN by various countries, giving an update about the work that is happening in that local setting related to issues such as poverty, hunger, health care industry, women, maternal health. All of these are part of the VNR process, but they tend to be quite large. Second, we want to look at a less traditional data source, which would be a collection of videos from something like in this case the AI for Good summit which occurred earlier this year. My hope is that by the end of this session you will feel comfortable connecting with large language lane large language models and understanding the various Wolfram tools that are available for that that you'll be comfortable with chunking and chunking data for batch processing, that we'll have an overview of how to structure calls and dip a little bit into prompt engineering, how to compile that final information into a resource that's useful for you, and give you a brief look at ways that you can expand those workflows to work in more complicated cases. Now, in our experience, we found three ways that you can work with external large language models within Wolfram. I'm primarily going to be talking about OpenAI's GPT model, but I do want to note that our team also regularly works with tools such as Palm from Google, Claude2 and Perplexity. All of these are going to require some sort of external account. I've included a link if you want to set that up for yourself. But the three ways that you can connect between a large language model and a Wolfram instance is the first is the built in functions. You're going to be hearing a lot about that if you haven't already here at this year's technology conference. On the other end of the spectrum is custom defined functions that allow you to interact with APIs via call and response, and then a blend between those two would be Service Connect. So a quick look at how each of those work. While there's actually 13 different large language functions that are that are within Mathematica currently, we're going to focus on just the SimpleTest, which is LLM synthesize. At its most basic level, you send a prompt to a GPT or a large language model and you get a response back. So if I ask what's the capital of North Carolina? The answer simply comes back as Raleigh. You can also be a little bit more creative in how you ask your questions and so Say I want to know about five different programming languages like Wolfram, but I want my output to be formatted as JSON and I want specific headings. You can request that and then work directly with that string. But there are some limitations. For instance, the default model in llmsynthesize doesn't work well with large chunks of text. If you wanted a summary of Alice in Wonderland, when you send that to the model, it comes back as an error. But there are tools built in that allow you to change the model you're working with. And this will come up later. But here we can say I want you to work with a much quicker and larger capacity model of GPT 3.5 Turbo 16K. And so here we get the summary. On the other end of the spectrum is using direct API calls, and I have to admit I'm a little bit biased on this one because a lot of our coding started this way, because a lot of the tools that we're seeing displayed this year weren't available yet. And so we decided to work with our own calls and responses. With the define function, I've included all of the code that we use. You're more than welcome to use it for yourself, but this is going to work very similarly. Our function is GPT. I can ask it a text. In this case, I'm going to ask it to summarize Alice in Wonderland again and I get the error back. But this time I can say, oh, that didn't work. Let me change the model to 3.5.16k. I also want to add something called a system message here. I say make sure that all your answers come back in the form of an eight line rhyming poem. And here you see, not only is it able to process the much larger text as before, but also we get the response in the way that we expect. A brief note on system messages in my opinion, this is the one killer feature of OpenAI's API that most aren't regularly utilizing. It was launched in March of 2023, and according to the developers, it's a way to pass on specific instructions to a model that is given a higher priority than the regular chat content. And so while often you can simply put your prompt information into the main call, using these system messages allow you to get a lot more flexibility to either add or remove some safeguards as you're working using these custom APIs also allow you, if you want to have the full chat experience of call and response that can be integrated as well. The last one, and I'm going to Skim over this quickly is the Base Service Connect option. You can open a Service Connect to OpenAI and then follow the syntax and ask similar questions. I've included this I actually don't know anybody who is using this interface regularly, but it does allow you to peek under the hood a little bit more. So maybe you don't want to interface directly with the API, but you do want to understand a little bit more about what's coming on with calls and responses. This is a good way to do that. And we get into a little bit about how you would send system messages, but I'll let you take a look at that later. I'll also mention we have a poll in this presentation if you want to let us know which of these methods you've used. I know I would be interested in knowing the answer, but that brings up the question which message or which approach should you use? We've used all three at various points. Personally, we lean towards the custom API interface because it gives us a little bit more flexibility, particularly as new arguments. As new API documentation is coming up, we can make sure that we're cutting edge, but at the same time it's hard to argue with something as simple as llmsynthesize prompt and have it get you your answer. For this presentation, we're going to use that basic function of llmsynthesize. That'll also make it easier for people to copy and paste our work. So let's look at a specific example. We mentioned the Voluntary National Reviews as a collection of thousands of documents where countries are giving updates on their work towards the global goals. And while hundreds and thousands of hours have gone in not only to writing the reports, but also the important work done to achieve the results, these are often overlooked simply because they're so dense. If we look at all the documents we have since 2015, there's 1300 reports. Many of them are hundreds of pages long. They come from 187 different countries and entities. And just as a personal side note here, if you look through the list of countries, you will not find the United States because we're one of only five countries yet to present a vnr, even though we are still doing some great work related to the Sustainable Development Goals. But looking at the recent year, if we want to know just this year, how are countries reporting? We have 175 documents that we would need to work through if we want to drill down into a specific example. Our neighbors to the north, Canada, they have nine documents that they turned in just for this year's report, many of those being hundreds and hundreds of pages when you look at them collectively. So what do we do when we want to summarize a document that in this case, as we bring it in, we see is 92 pages long? Most of the demonstrations that I've seen from various large language model demos tend to look at very we call them cats and dogs examples. Things that are very small are going to give you predictable results. But what if you want to deal with much larger sets of information? We know that the GPT models that we've worked with simply can't even come close to handling 265,000 characters. So we know that if we want to be able to work with it, we either need to condense it, we need to chunk it, or we need to discard part of it. And so we've started integrating a few tools to be able to work with these large strings. The first one is very simple. It's a token length approximator. In general, at least for OpenAI, tokens vary in actual length, but if you divide by four, you generally have a good idea of how many tokens are in a string. So we've taken that ability to estimate and combined it with one of Bob's functions from the function repository called Sequence select, which essentially selects from a list up until the point that it reaches a certain test and then continues to process the results. By using that initial code, we've come up with a function called Token Part Split, where we can import individual pages of a PDF and then it'll automatically chunk it as close as possible to a maximum token length, so that when we send a prompt to GPT, we can know that we're processing as much information as possible without going over that limit. And so if, for instance, we wanted to work with the most powerful model, the GPT4 model, this is going to be slow, it's going to be token limited, but it's going to be a little bit more precise. That works at 8,000 tokens currently. And so since we need to account for both call and response, we can chunk at 7,000 tokens. And we see that this 92 page document can be broken up into roughly 10 equal length segments that can then be processed. But if we're willing to go with the faster model, not quite as precise, we can use the GPT 3.5 Turbo 16K. It's a mouthful, a little bit less creative, but for summaries it tends to work very well. And here we find that because we have 16,000 tokens, you're getting a larger context of information and we can take this 92 page document and come up with five chunks of information to send to GPT. So we're going to look at it two ways. Let's start with GPT4. This is our 10 chunks that we have. And this gets into the prompt engineering. The first thing we want to do is give the model some context for what it's looking at. We're saying this is Canada's voluntary national review. This is when it was submitted, here's the chunk from it, and then we give it instructions. We want a bulleted list summarizing the content and then list some specifics that we want it to Pay attention to. 10 chunks. 9 minutes faster than reading a almost hundred page document, but still a bit slow. Now when we switch to the 3.5 option, we're doing the exact same process, but now we're only sending five chunks. Same questions are being asked. And here we're able to complete the process in about a minute. In our experience, and we've done extensive testing with this, we find that for summarization 3.5 Turbo does a more than adequate job and the increase in speed and token abilities make it worthwhile for using. And so here you see that we've ended up with a Response that's about 3,000 tokens long. And we have a bulleted summary of everything that is included in that document that by itself isn't as usable. So let's now bump it up to, to the more sophisticated GPT4 model. Here we take those bullets, we send it to the model, we tell it what it's looking at, that this is a summary of the, of the document and we ask for a, a one page written summary. And so here we get a, in the span of, I don't know, three minutes total, a 92 page document that, that covers the entirety of the work and details, including included in it. For me, the biggest takeaway of this is that by doing it through this methodical process, you know, every step of the way what assumptions are being made, what information is being lost, what prompts are being used and, and also how the, the information is being handled. Now if we wanted to take that, that same idea, I gave you a very specific workflow example for, for Canada's vnr. But what if we wanted more of a all in one PDF processor? We can take that same workflow and with it do a few changes to it and include a few tricks that we've learned along the way. The first thing we want to do is go with a more generic prompt because maybe one day you're working with a UN document and the next day you want a summary of a work related PDF. So we make the prompt more generic. We use the file name to provide context because often that does include some information. The big shift in output is that instead of asking for a string, we request that GPT gives us its output as HTML. Not only is this easier to read, which was our first foray into this experiment, but also we found that GPT in general does a better job of classifying and categorizing and chunking the information that it's giving back to you because it's able to use headings as part of that process. The other thing we've done is we've already done the hard work of getting bullet points for a PDF to summarize. We want to make sure that we retain those for going back. We use an algorithmic process to turn all of those bullets into a UL li HTML input and then we export the entire thing as a file. We've already done Canada using now all the lessons learned. In the Canada example, Putting it into one line with our PDF summarizer, in about two minutes we can summarize Belgium's VNR 130 page document. Even as I'm scanning this right now, I see that next to last major heading measuring progress and SDG indicators. My eye is going to go to that and I know that that's something that, that I want to pay attention to. This is just a screen grab, but you can see that you also have all the bullets. If you want to get a little bit more in depth, we can do the same thing with the European Union's VNR which this was the first year that they presented. And in this case we realized that this is a 300 page document with over 250,000 tokens and it requires 19 different chunks to summarize. So this is starting to push the limit of what we're able to do with our code. But in the span of about 4 1/2 minutes we get that same overview of all of the progress the European Union has done towards achieving the Sustainable Development Goals. Now a few notes on this process. Admittedly we, this is pushing up against the very edge of what we can do with our token limits. We have other tools that we're going to demonstrate in a second that deal with intermediary process. So maybe you chunk, you summarize, then you have to condense and then you summarize again. That's Something that can be can be built in. Second, this process is built on the assumption that, that you already have a large string. Pre divided PDF pages, as we said, works very well. But you can also do something as an alternative approach and do something as simple as string splitting based on line breaks or even just a simple string partition will get you where you need. I'm sure some of you are sitting back saying, wait, aren't there already tools for this? Why would I do this for Wolfram? And I think that's a great question. And yes, there are tools, but my question back is when you use somebody else's tool to summarize a PDF, whether that's the Chat GPT interface or some of the other functions that you can work with, or standalone websites, do you know what's going on inside the black box? Do you know what pre processing is happening? Do you know whether the entirety of the text is being considered? Do you know what prompts are being used? Do you know where, what models are being used and at each phase? Because in our experience, anything that if you can get a 300 page document summarized in less than a minute, then there's some trade offs that are probably happening along the way. Give it six months and that probably won't be a limitation given the speed of innovation we're seeing. But for me at least, I like knowing what goes in, what comes out, and what trade offs were incorporated. So given that that was kind of a practical walkthrough of how to work with large documents, as well as some code for implementing some of those processes, I want to shift gears a little bit and demonstrate a few of the team tools that we've used. All of these are based on the processes that we just discussed. The same idea, the same workflow of chunking, summarizing, recombining and then exporting in a usable format, all of that is being used. These have been a little bit more customized to our own team's needs. And so I'm not going to share the code with these in this setting, but I did want to show you what was possible with just this mindset or just this work process. The first would be the question of what do you want to do? What do you do if instead of just summarizing one large document, you want to summarize a large set of documents? So if we go back to our VNR example, in 2023, there were 39 different countries that submitted what if we wanted an overall summary of all of those VNRs? What are the main messages that are coming from those VNRs. We've created our own custom function called FileLists Summary that will automatically batch process that entire list of files. We originally used this for a folder of transcripts that we had that we wanted to get a summary of. But here you see it also works with large number of UN documents. You could use this for technical manuals or anything else. And so here, 39 documents and about 10 minutes we have an HTML summary explaining the main Messages of the 2023 Voluntary National Reviews. We've also included a data set or it actually comes out as an Excel file where all of those summaries, all of the original bullet points are maintained. And so if you want to know about about Barbados main messages, you can go in and read specifically those bullet points which might not have come out in the, in the short summary, given how much information we're starting with. We also keep all of our intermediate notes as GPT is condensing. We want to make sure that we have all of the different levels of information that was created. We actually wrote a medium article on this process and about how we used the Wolfram tools and how we used large language models to create a summary of the voluntary national review process. Because it is so dense, a lot of people were just kind of skimming over it. We wanted to be able to demonstrate going a little bit deeper. That article is linked there at the bottom. Global Goals and AI Solutions. A different example, going to a more traditional data source. What if you want to work with a YouTube playlist? Let me set up a theoretical use case for you. There's a conference that you were supposed to attend and they've posted all the content from the sessions onto YouTube. When you get to the end of the session, you realize that there were a lot of important sessions that you weren't able to make it to. How do you get to a spot that you can go back to your colleagues and give them a summary of all of the most information or all of the important information? Well, the first thing you would need is a tool that automatically imports all the video names and URLs from YouTube. And so that's something that we have in our, in our toolkit. Next, you'd want a tool that would be able to extract all of the information from individual videos. And so here we're looking at the first video. This is a keynote speech from the director of the International Telecommunications Union at the AI for Good Global Summit. We see who created it, who posted it, the date that it was posted. We get the SNIPPET we get the full description, the meta information for the video. If the creator decided to include chapters, we have all of those that we can work with. We have a transcript list which includes timestamps and as you can see when you scroll down, it also brings in the textual data needed to process in a large language model. In this case, the full transcript of that information. In this case, this was actually a conference that we were not able to attend live or able to follow live. We ended up, I think we had about 30 or 40 hours worth of content, content that we wanted to catch up on later. And so using our Playlist summary tool, we tell it what the language model is working with, we give it a link to the playlist, it automatically goes out, pulls in all the information from YouTube and then does an individualized summary for each of those videos, and then also gives a full HTML summary of the event as a whole. Each session has all of the metadata related to it as well as bullet points. So you can figure out which one you might want to watch on the plane heading backwards, as well as all of the intermediary steps that we have already talked about. And as one final example, and then we'll turn it over to Q and A. In yesterday's presentation from Maureen, she talked a little bit about how we've done work on gathering news and web articles, filtering them to make sure it only includes the most trusted and unbiased sources, and then auto generating articles that we can use to do further fact checking. She shared a bit about that. And all of it is at the end of the day based on the same processes that we've talked about in this presentation. But we've even taken it one step further that I wanted to demonstrate to you is we've made this into a web based interface for our team. And so any one of us, it's not only Maureen and I, we also work with a marketing and business expert named Kinet who is able to use these tools as well. You're able to come in and say I'm interested in this search term and this idea. You can choose whether you want just web results or news results, or a combination of both, how many different resources you want to bring in. And then you get your auto generated article. Here you see. One of the other advantages of having HTML as the final output is that when we see Stephen Wolfram's name at the very beginning, we can actually click on it and and go directly to the blog where he's talking about these things. Just as a side note, because I don't know who from Wolfram is listening. I will say that suspiciously as we came into into this, this talk this week, that we are timing out when we're trying to do this process online. But it worked last week. But again, I started with the caveat that things are changing daily. So just to wrap things up, as we've seen, LLMs and AI tools are changing the way that people work. We have the ability to summarize and to think and to process items that we never would have been able to do. There's no way that people have the time to read through 1200 different voluntary national reviews, but with processes like this, we're able to get those summaries. 2. We are strong proponents of the Wolfram technology stack and we think that it is more than just a pass through of these LLM tools to connect you to external resources. In fact, a lot of the textual and procedural functionality available within Mathematica allows you to easily prototype a ton of tools. And also, and I think this is most important, know what's going on inside of the black box. And I also have to say that while a lot of these implementations, whether it's direct tools or functions being added by OpenAI, while new ones are being released daily, we find that the ability to customize based on our own needs and constraints, constraints will always ensure the need to have these homegrown and tested approaches. But as we've mentioned before, we expect that this rapid change is going to continue to require developers not to do less, but to do more in the way they think. So that is our presentation. I think we have a couple minutes left for Q and A. Maureen will join us for some of that, but please submit your questions to the webinar. Okay, from John, what's the bigger news here? The positive value of using your tools or the ability to condense and distill valuable snippets from the big Internet ocean? I would say it's both. I would say that the tools are going to constantly be changing, but the thinking about it is what the big news actually is. How are we processing the larger amount of information that is now available to us? The VNRs, I think are a good example because it's all very important data, but it's too big for any one person to be able to keep up with. Even if you did read it all. It's hard to keep all that in your mind. And so I think that the distilling, the ability to work with larger data sets at the same time is a huge part of what's happening. And I think Maureen and I were talking about this earlier today. The changes we've seen in token limits. We started on GPT3 and we had 2,000 tokens. Then we moved to 4,000, then 8,000, then 16,000. And there are some models now that can go up to 32,000 tokens. And so I think that as those technologies continue to advance, we'll continue to see more of that. Anton asks special LLM tools that are implemented by you do interface to markdown or latex. Most of what we do is is directly through the API so we can control how it's being processed. We haven't used LLM tools, we've used very few LLM resource functions. And the reason is is that we're kind of nitpicky and we want to know exactly what's going on behind the scenes and we like having that control and it's been easier for us to do that individually. So Bob who I hope you were here early enough to see your shout out for your Wolfram function repository entry. You ask about the practical use of this tech and what other tech from Wolfram would you like to see?

Wolf wow, that's a good one.

Yeah, jump in Maureen, I'll let you.

I would like to see. My number one wish is actually for what's in WolframAlpha and in the entity database, both for more current information and more comprehensive information. Because using Wolfram to check facts is very important for us. We, we do as much fact checking as we do gathering and Wolfram is set up to be perfect for that. Sometimes we can't find the most current information or the most comprehensive information that it would be great to have.

Yeah. And I think that we focus largely today on the non LLM tools for processing text, which I think Wolfram is excellent for. And we've seen a couple demos of using some of the entity data. But the resources between entity stores and other interfaces I think segue very nicely. Other tech from Wolfram for me it continues to be the transparency piece, not only of what's happening behind the scenes, so that it's always easy to know exactly how back and forth between the models is working, but also transparency in terms of how. How do we know what's going to happen with new developments as APIs change. You know, if we're coding in one way and a new endpoint is released next week, that's something that, that I would like to know more about. I do know there was the session this morning on the service Connect changes which I will definitely be checking out because from my understanding is is that that might answer some of my own questions. Okay Timothy, comment about tech summarize new in 14.0 we will definitely check that out. Is that can somebody get back to me and let me know is that an LLM based function? Does that require the same back and forth? If it is my my question would be I would want to know the processes going on. I think that for most users having a single line, single function, beautiful, elegant tool makes a lot of sense for other users, particularly for more other users who have a need for a certain level of data integrity. I would want to know what's happening behind the scenes.

Bob I keep thinking of things I would love to see I think in the prompt repository. So some way of knowing what the quality and depth of the prompt you can get talk like a pirate really close to something that's really super useful for people. I don't know how to implement that, but would love to be able to use the prompt repository more but find that it's often easier just to write my own than to find stuff still.

So Sid makes a comment about token link limitations. I have a strong feeling that when we're back here next year that a lot of the information and processes that we're talking about will be significantly outdated because of the increase in token limits. Maureen and I have also talked extensively about some of the multimodal processing that is available through the current GPT4 model, and then obviously any new models that will be trained between them. My response to that is that I think that the thought process of being invested and intentional about what is going to what model, how is it being processed, how's it being transformed? The more steps you can know within that black box, the better off all of us will be.

Chat GPT in the Pro Advanced already is up to 100k tokens. Perplexity has a higher token limit and Claude 2 has a higher token limit. So if my trade off is between fast or working through in the way that Ben just described, I'll often do a cut and paste and then do a summary in Wolfram and I think all of the API for some reason the Chat GPT API is lagging. What they're showing in their own products. I guess we know the reason for that, but it's all going to be there pretty fast. Once they combine their different betas into one API, which I assume they'll do and have people pay for, it's going to all be doable through Wolfram.

Well. And since I know we have several Wolfram people here, thank you for all of your hard work. Thank you for giving us a platform that we can build out and experiment with. I've really enjoyed this year's conference, seeing how some of the same problems we've encountered in the past six to nine months have been solved by the team there. So keep up the good work and thank you for the presentations you have to participants who are still around. I would absolutely encourage you to check out the other LLM related talks from this conference because they really are top notch.

And I'll add my thanks for 10 plus years of playing with Wolfram because I'm not a coder and with Ben's help, I can still use it for all sorts of things. Things. Also, it's very helpful when we're trying to pilot things, the fact that you give us all that access to what's under the hood so we can get things out early. Because we're very interested in using Wolfram for our production work.

Thank you all. I don't want to keep you too long from other people's excellent presentations. So have a great day and hope to see you all in person soon. Cheers.

Thank you.