Using the Wolfram Stack and LLM Tech to Combat Misinformation on the SDGs and ESG

Good afternoon, I'm Maureen Baer and I'm here with Ben Kikert to do our presentation on using the Wolfram Stack and LLM technology to combat misinformation on the SDGs and ESG. I'll be doing the first part of the presentation on video because I have a lot that I want to say and want to make sure that I don't ramble, but Ben and I will be available after at the 20 minute mark to answer any questions that you have. We're a social impact organization and we have two main areas that we've been concentrating on for the past several years. The first is about gathering and curating good information for startups, impact investing and social change, and sharing that information with appropriate audiences to help them have more impact and understand the field. The second is following and championing data and metrics for the UN Sustainable Development Goals. And this year we've been platforming for a third which is combating the potential of LLMs and other AI to be used to spread disinformation, especially in the run up to the 2024 election process. We take a journalistic, fact and evidence based approach to all of our work, but we are advocates, we do have a point of view and a mission. For example, we believe that startups with intentional social impact embedded in their organizations are good. We we believe that individuals, funds and others investing for social impact is good. We believe that trying to accomplish the SDGs the Sustainable Development Goals is good. We believe that keeping track of and encouraging good metrics and Data for the SDGs is good. We think that AI and other LLM tech has the potential to be very very good or it can be bad. We don't know yet. But we do know that if you don't play you can't win. So we're in the game full throttle. We believe that deliberate misinformation and disinformation spreading is bad. And we believe that the growing and growing information glut has potential to be as bad as disinformation because it gets harder and harder for people to differentiate information or even find it. Sometimes there's too much. And a bit about us and our work styles. First of all, there's only three of us and we all have other work so we don't do this work full time. Kannette is our editor marketer and starting up good is her primary content expertise. Ben does all the code, all all the tech experiments, spends lots of time educating me and Knet on Wolfram and his expertise is SDG counting and all things data. I do a little bit of everything and have lots of ideas. Sometimes too many. The second thing is we're very outcome focused. We have a mission and a purpose and all tools we use are to support that and to extend the reach of our very slim staff. We look at all tools and make all decisions based on how we can use them in our end products and what they can do for our mission and purpose outcomes. And we like to try new things, experiment, and we like to do them in a garage startup style. We've been early adopters of all things Wolfram and Explorers in various kinds of AI. As a matter of fact, we did, I think, the first presentation on large language models at last year's Wolfram conference, and we got a pretty early start in terms of large language models and building off that, we started incorporating tools and bits of tools into our work quite early. For the rest of the talk I'll be giving you a high level view of some of the code we use in our daily work and in tomorrow's presentation Ben will be doing a deep dive into some of our code and tell you about how he's coded it and the kinds of choices he's made. But first I wanted to give you a bit of a sense of what we're using the code for and what our output and product is. We write medium articles and do social media to publicize them while interacting with other people with similar interests in social media. In 2023 so far in our two properties SDG, counting and starting up good, we've published over 50 articles. We do social media for them on Twitter, LinkedIn and sometimes Facebook and Instagram. We were big Twitter users for social media until it became X, and so we've shifted our emphasis to other ways to communicate with people and tell them about our articles. Each of the articles that we do is very well researched, and we use LLM tools primarily to gather research and do summaries. Our authors and editors, Kannette Benn and me go back to primary sources on pretty much everything to make sure that there's been no hallucination or other funny stuff in what LLMs have given us. We've had lots of back and forth discussion among ourselves on whether to use generated content from GPT4 or not. We've swung in both directions over the course of the year though, never publishing anything that an LLM wrote, and we've decided at this point not to use LLM tech or tools to actually write for publication. The toolkit coded and constructed by Ben is the heart of how we use Wolfram and LLMs to do our work. It's been built iteratively over the past year based on what we need for our products and the outcomes that we want. Also, Ben has coded many functions and super functions in anticipation of our needs. At this point there are over 50 functions and super functions within the toolkit in several different categories analysis data sets and lists, article workflow functions, pure GPT, GPT Chat and Dall E connection functions, code for specific sites, and some general web functions. The toolkit also includes explanations and examples for each of the code pieces. It's 147 pages in PDF and growing. Now I'd like to describe a few of the tools that we use in our daily work. Every morning we get an email that provides us with an overview of the newest news on our topics for that day. The way that this is constructed is it's run daily through Wolfram by BennerConnett using as base our paid subscription news aggregator service. It's divided into our six core SDGs, startups, impact investing, ESG, deep learning, and Virtual Reality. We get articles in and then they're curated through Wolfram based on our Trusted Sources list, which is about 40 sources out of the 200 plus that can come in through the news aggregator. This info is chunked and sent to OpenAI GPT4API for summarizing by topic. You'll see the summary at the top of the illustration on the right. It's then formatted appropriately for email and sent to us. All of this is done through a Wolfram super function so that we start every day knowing the most current news and information for the day. Autoarticle HTML is the core research package we use as we prepare to write our articles. It's a super function and it's composed of eight sub functions in a module. All I as the user have to do is enter the title of the article I want to write and push a button. It takes three to five minutes to run, but it's worth the wait. The first part of the module Search Terms takes the title I put in and asks GPT to give back several prompts that then can be used by web Search to bring in source titles, snippets and URLs. After reformatting, Web Search returns 10 results for each query. From Bing Search, the results are deduped and reformatted to go back out to GPT for article drafting. Article sub function is a prompt that includes the raw results as well as a long system message to tell GPT what to do, in what style, and perhaps most importantly, what not to do. Constraints to GPT are important. Once this raw response comes back in, it's formatted into HTML, the sources, prompts and source result lists are added back in and it's exported. I as the writer now ready to select and read full length articles, check sources, et cetera. It gives me a good foundation, comprehensive and organized. Here's an example of the output from autoarticle HTML where The title is ESG News October 2023. The article draft is on the top left, the sources bottom left. All of the links are live on the right. You have the search terms that GPT came up with to feed into web search, and then there's a long list of web results, only showing a few here, but there are 50 in total we're always in the process of incorporating new sources and super functions into our daily workflow. Recently we've incorporated Bing News Search API because it has a freshness date that allows us to get more precise current news than web search can. And based on Steven's keynote and other presentations this week, we expect to be able to add many more external service resources as well as API connects to other large language models to our tools over the next several months. Another tool that we're very excited about incorporating into our super functions and workflow is AdFontes, a tool for reliability and bias ratings. You'll see the diagram illustration of what it does here on the screen. Essentially, it goes through 700 media sources and through a pretty rigorous analytic process, ranks them in terms of their reliability, their factualness, and their bias. Are they neutral in the middle? How do they skew to the right or left in terms of their perspectives on politics? We've downloaded and structured the data to run our source articles against and use it by hand to allow us to screen in and screen out sources for some of our other super functions. We find it equally important to make sure that we're seeing what all media bias sources are saying. If you only read and pay attention to and write from your own place in the quadrant, you're never going to be effective at influencing and advocating for your mission in other places. We see this as particularly important as we begin our work for paying attention to the run up to the 2024 election. This has given you a glimpse of how we use Wolfram and large language models to do our work. As you've noticed, most of our code is not through the elegant structure of the wolfram functions for LLMs. That's because we started using our own code, hand grown by Ben and have proceeded on that route. He'll be telling you tomorrow a bit more about why and how various things are integrated. So what's next for us? First of all, we see ourselves using many more super functions with Wolfram as the backbone, While integrating more LLMs, Palm Perplexity, Anthropic, Claude 2 and other outside services for finding the raw material for our articles and implementing our social media. We see increasing use of web search, Bing search, Google search, etc. To do fact checking, score facts and make sure that there's no hallucination coming through. We're planning to incorporate advantage reliability and bias ratings and also use them to build some small specific topic sentiment neural nets as different kinds of LLM models appear and models are upgraded. We want to use more than OpenAI, but we also see an upcoming problem of which ones do which and how you discover emerging truth and bias issues. As the terrain of LLMs is changing so quickly, there's still black boxes, but we're doing everything we can to tease out how the various black boxes think reason and put out one word at a time. And for our mission work, we plan to apply all of this to the upcoming United States elections. We're concerned about them. We're concerned about the glut of information that's available. It's increasingly hard to separate the good from the bad, the neutral from the biased, the points of view. It's increasingly hard to penetrate the social media ecosystems that don't represent your views. We don't know what we're going to do about this, but we do know that based on how both our own expertise, tools and points of view have changed in 2023 and how quickly the field is changing, both the opportunities and the challenges are likely to be even greater in 2024, and we look forward to continuing in our 10th year of using Wolfram as our fundamental tool and the backbone for our work.

So Maureen and I are both in the chat here. If there are any Q&As that people want to submit, we would be more than happy to answer those. Let's see, there was a question already about the handouts and after this presentation we'll upload documents.

Hi Anton, do we think this information should be battled with this information? I think that Ben is going to know a little more about what this means than I do. When you say battle, I'm not sure what you mean. Ben.

Yeah, I'm wondering if his if his question is should you combat misinformation by pushing out more correct information. And I can say that that's a philosophical discussion that we've had quite a few times as a team, particularly looking at what is available from AI capabilities. As Maureen said, it's not something where we have started publishing, but knowing how much negative intentional misinformation and disinformation is out there, and knowing how LLMs are being trained on the full corpus of data on the Internet, I think it's a question that we need to consider and keep considering in terms of do we need to make sure that there's enough good information for training, knowing that there will be some bad information as well?

Anton, I see further down you put disinformation versus disinformation, I'm pretty sure that we're going to combat disinformation with the best truth and reliable information that we can. I understand what you mean and sometimes it's very tempting, but I at least am a information pacifist at heart.

Bruce, I see that you asked about the the company name. We tend to refer to our work more as an initiative because we are working in more of a voluntary and part time capacity. But the two initiatives that we work with starting Up Good is the one that focuses on entrepreneurial support, working with startups that are interested in doing social, environmental, societal good. And then SDG Counting would be the initiative that focuses on tracking and sharing and advocating how sustainable development data is being used and being shared. There's a question at the bottom, how does the interpretable AI would be used as a way to aid the legal system? I would say that that is outside my scope of knowledge.

Yeah. What I can say is there's lots of tools that are built on top of I can speak to ChatGPT, particularly outside tool builders who are doing things around the legal system and it might be worthwhile for you to explore Perhaps actually using ChatGPT to find out where some of them are. Browse around. What I would say is until you're sure of their reliability and their bias, don't put them into operation.

And just from a legal perspective, the Wolfram attendee, Seth Candler, who's here, I know he's got a presentation later on his book on queries. He works extensively in the legal domain and so while I haven't had a chance to talk with him about how these tools could be used in a legal framework, he would probably be the first person that I would call. Looks like we skipped over a question. It says Wolfram language seems like a natural choice for this type of Analysis. What would you say that the language could do to make this analysis even easier? New features, particular areas that were more difficult than others to develop. As Maureen had mentioned in her presentation, we ended up doing a lot of code de novo for our own purposes because we were trying to work ahead a little bit from where Wolfram had implemented their own LLM features. We'll go pretty in depth tomorrow during my presentation looking at some of the tools that we use. For me it's the pre processing, particularly when you're dealing with large text samples and you want to be able to make sure that you're including all of the information. I think that all of the textual and string tools that come into Wolfram make it a very natural choice. And we've built a lot of our tools. Yes, we're using AI, both GPT as well as Palm and some of the other models that are out there. But at the end of the day we're using Wolfram as kind of the framework for all of our calls and structuring. I mean, there are other languages you could do that as well. But I think people get a little bit too excited about what you can do with a one sentence prompt on ChatGPT, when in reality it's the back and forth that really gets you a better return for your data investment.

And one of the other things that I'll say is that we try to use various fact based sources or as Stephen Wolfram would say, computational data and also web search to layer GPT and sort of surround it with things that are facts and are authoritative and we feed it. I think it's important that you feed GPT nutritional reliable information and then tell it not to make anything up. We find that we get pretty good results from that. But in my presentation, one of the things I was trying to say is that the role of the human gets even more important because as these new models are evolving and new features are evolving and the world of information is getting more messy, you can't beat a human for discernment.

So Anton's making a comment about some of the cross referencing and looking at propaganda. This is one of the things that I imagine we will continue working on and continue sharing our work. And I think it goes back to the question about where Wolfram language can be most useful. And that's to say that I think that from a cross checking, fact checking point of view, that actually there are quite a few tools that we're ready and excited to implement. And to look further at, we talked a little bit about looking at some of the source information. We didn't go into it in depth, but there are good resources in terms of understanding bias and reliability ratings. But even beyond that, one of the things that we've looked at implementing as well is some of the knowledge base that comes from Wolfram. Looking at the entity stores, the different repositories as a way to be able to feed more information in Wolfram Alpha would fit into that as well.

And we're hoping that as the Wolfram plugin is used more and the computational and reliability factors of Wolfram Facts are used more, that there'll be additional categories added and Wolfram will continue to have the most reliable information and up to date information out there. That's a plug to update the information on the UN please.

Yeah, I saw that Marco had mentioned some similar work going on at a website. I appreciate you sharing that and we will take a look at that.

And Anton would love to have another discussion with you about some of the ideas here because I think we're working. I'm not sure whether in parallel, but we. There's some cross fertilization to be done.

I think that gets us to our 1pm yeah. Thank you all for the questions. Thanks for the feedback that makes it all the more fun.

Thanks very much.