Computation and Chemistry: Live Interview with the Academic Innovation Support Team

Foreign.

Thank you for joining us today. I'm Noah Chartoff. As always, I'm joined by my coast, John McNally here. We are both Wolfram employees, education backgrounds, myself in mathematics, and John primarily in physics. We're joined today as well by Jason Sonnenberg, who is our head designer for chemistry educational materials here at Wolfram. And, of course, we are joined by Joshua Schreier, who is here to talk to us today about a lot of things that he is doing with Wolfram technology in the study and teaching of chemistry. So, Joshua, why don't you tell us a little bit about yourself, introduce yourself, let us know your background, your research, what you're into, just anything you'd like to tell us.

Absolutely. Thanks. And yeah, I'm really pleased to be here. You know, I enjoy using the Wolfram technology for lots of different things, both in the teaching lab, in the research lab, and just for fun. So, you know, just to give you a little bit of background on me, I'm a theoretical physical chemist. So as I like to say, that means that I really only understand two and a half things. Quantum mechanics, thermodynamics, and kinetics is kind of half of a theory, right? We could derive it from the first two, but what we do is that we use our understanding of those physical theories and use computer simulations to help us simulate chemical phenomenon, Right? And these could be chemical phenomena related to how materials, the structural properties of materials or their reactivity or how they interact with light or with electricity. And so my background is doing that sort of traditional computational chemistry work where we simulate things. But about seven or eight years ago, I got involved in machine learning and in particular, trying to develop machine learning models that were helpful to people in an experimental laboratory that were trying to synthesize new materials. And so that's gotten really popular. And so that has kind of taken up a lot of my work. If you do the Google Scholar sort by date, you can see that the kind of all the recent things that I've been working on have been kind of focused on this area of machine learning now.

So for viewers of ours who might be more familiar with chemistry than machine learning, what does machine learning mean in this context?

Yeah, so, I mean, we use lots of different types of approaches. The early work was really sort of supervised machine learning. So we were interested in this problem of given a chemical recipe, right? A set of ingredients, a set of reaction conditions and amounts, can we predict what the outcome will be? And you could think about a hierarchy of types of things that you might want to Try to predict, I mean, the lowest level on that hierarchy is just like, thermodynamically, can a reaction occur? And then higher up on that hierarchy might be, well, what are the types of structural arrangements that result or what are the properties that result? And so our beginning forays into this were really motivated around how do you get data about experimental work? And it turns out that a lot of that data is just not published. Right. I mean, all the failed reactions and even the marginally successful reactions are just not present. And so, yeah, I've heard that referred.

To as the file drawer problem, that the only things that get published are the ones people found really interesting. And maybe there's a lot of data that shows a lot of things that didn't actually make it into the field.

That's right. And, you know, if you think about using this for a supervised machine learning approach, right. If you just provide a series of positive examples from the chemical literature, right. This recipe worked. This recipe worked. This recipe worked. Well, it's very easy for your supervised machine learning model to then just say, well, everything works. So what we did back in this paper in 2016 was essentially we paid undergraduate students to go through a decade's worth of old laboratory notebooks and digitize them. And then we built what are, in retrospect, very simple machine learning models, just simple SVM models, the type of thing that you could do using the classify function in Wolfram language. And we found that those very simple models were actually just as good, if not better than expert chemists at figuring out what reaction conditions to try. And the thing that we did that was really kind of fun was that we closed the loop on the process, which is to say we actually had students go into the laboratory and perform reactions, reactions that human experts have specified and reactions that the machine learning model had specified. And so that's become now something that is a lot more common. But when we got started, I mean, we were really just kind of fumbling around, and even this idea that you needed, I like to call them the dark reactions. Right. I feel like the physicists have all these great names like dark matter and dark energy, so we need a little bit of that spice. But, you know, the value of those, those sort of unpublished failed experiments, right. That, as you said, you know, otherwise linger in a file drawer, but are really valuable for improving the quality of your machine learning models. So, yeah. Oh, go ahead.

I was going to say the machine learning effort reminds me of the expert systems that were so popular in the pharmaceutical industry before machine learning existed. And they were, you know, trying to predict based on a variety of collections of data.

Yeah, that's right. If you think about those efforts, I mean, go back to the late 1960s and early 1970s, right. To sort of the beginning of that sort of good old fashioned AI. And so this is a field now that has become really well established within the American Chemical Society and the Materials Research Society. Right. This idea of data driven chemistry and materials research. And so it's been really fun to kind of grow up alongside that field. So getting back to your question, Noah. So I mean, that early work was really about sort of supervised machine learning. The types of things that you would use, using, for example, predict and classify in the Wolfram language. More recently we've been experimenting with different types of generative approaches. Right. Things like generative adversarial networks and kind of other generative models for trying to design experiments. And so that's sort of work in progress on different fronts.

Interesting. I think within the last year a lot of people have gotten very familiar with generative AI like ChatGPT. We're used to AI that creates something, but. But it sounds like a lot of this was about getting AI to for one thing, understand or at least categorize things and also to predict based on that information.

Yeah, that's exactly right. So I'm actually on sabbatical this year. So this is on the left is what I looked like before the sabbatical, and this is what I look like during the sabbatical or maybe just right before the sabbatical. But kidding aside. Yeah, I mean, generative AI and in particular things like large language models. Right. Have become tremendously popular. I mean, there are articles in the New Yorker magazine or the New York Times about these types of things. So it's reached a broad audience. But they're also an emerging field of research in science. Right. This is actually kind of a recent open access paper that a whole bunch of people did, right, where they had a hackathon on how do you, how can you use LLMs to, to maybe transform material science or chemistry research? And so there are a lot of really neat examples that, that are in there, including my former postdoc, Kwin Chong. I, right. Is the second author on there. Really kind of neat, neat, neat story. You know, I have to admit, you know, it's not something that I really have played around a lot with until this past summer, right. In August. And so one of my goals during the sabbatical is really to play around a bit more with large language models. And these other types of emerging generative approaches, try to understand them a bit more and then also see how we can apply them to chemistry. And there are people that have started off these efforts, right? There are a lot of kind of nascent things that are out there, but really a lot of exciting opportunities. And so I think the explorations that we're going to talk about today are in part, you know, my efforts to try to get up to speed on what people are doing and then try to put them to use right. In problems related to chemistry, problems related to my own chemistry.

One of the things I like about some of the community posts that you've done recently is that it sort of touches on, you know, both things that people can mean when they say using large language models for science. So one thing they could mean is using things like GPT4 and other things that are pre trained with a lot of language and so they can help with the coding process. But then there's also this other question of, you know, can some, you know, large transformer based model that doesn't necessarily have anything to do with natural language, but could that also be helpful just as a machine learning tool for these research areas?

Yeah, yeah, that's right. And you know, and I think it's really kind of fun to kind of explore both of those, those angles, if you will, you know, because certainly, I mean, you can think about lots of problems as involving linguistic tokens of some kind or, you know, but those linguistic tokens are not necessarily the same ones that, you know, ChatGPT has been trained on. And you know, perhaps another thing that comes to mind in this effort, you know, there I can dig out the link later on, but some, some folks at the U.S. department of Energy actually have a really nice website and report on AI for, for science, Energy and security that just came out this past fall. But you know, that report really unpacks, you know, some, some of those, those different ways of, of applying generative AI to, to science problems. And they range from, you know, maybe we need to train foundational models on scientific literature, or maybe we can train foundational models on numerical simulation codes that are used at the different Department of Energy labs. But also what are maybe fundamental algorithmic ideas that can be repurposed and developed to really help us understand matter and energy at all of its length scales.

Very interesting. I hope that they're taking into account the sort of thing you were talking about before where published literature may not be the be all and end all of data that represent the actual world, but it does sound like a fascinating project. By the way, when we're referring to posts on community, we're talking about community.wolfram.com where Joshua has some fantastic materials and a lot of other people do too. By the way, if any of this stuff in the notebook is going by faster than then you are following, don't worry. We are going to post this whole notebook linked to a permanent version of this interview in the YouTube location that it's going to be once it's over. So you'll be able to return to this later on.

Yeah, and thanks for that clarification. I mean, actually, I mean, maybe this is a good chance to just pivot into some of those, those community posts. Let's do it. And thinking about these things. So again, as Noah was saying, you can kind of find this post on community.wolfram.com this was a community post. I had actually written it up as a blog post and it was really an answer to a Mathematica Stack Exchange question that was posed. And I love just hunting the Mathematica Stack Exchange for interesting questions and, you know, a chance to sharpen your, your sword.

I'm a huge fan of the Stack environment as well. I'm, I'm on there every now and again. It's, it's just a great community.

And so user5601 had a, had a question, right, where he or she was asking, you know, whether there was a way to, you know, get the OpenAI API to summarize documents that were longer than the input limit. And this was actually something, I guess this was back in August. And so the input limit at that time was about 4,000 tokens. Right. So kind of roughly speaking about, you know, 2,500, 3,000 words. And so then the question is how can you kind of expand around that? And so I guess just as a useful note for the listeners today, you know, certainly that that context window has expanded a lot.

I believe it's around 16,000 now.

Yeah, yeah, that's right. And there are even, you know, sort of, you know, preview versions that are like at 32,000. Right.

But even with three four page documents, I've run into the limit. So I find this sort of thing still very relevant.

Yeah, right. And so, yeah, I mean, it just kind of increases the size of the chunk that you need to start working on. So this was something again, as just somebody that was trying to learn this field. I didn't really know any of these methods and so but I thought that given all the great LLM functionality in Mathematica 13.3 that I could figure it out, right? And so I did a little bit of research and tried to find out what people knew and found this post by Isaac Tam, right, that had been written a couple months earlier where Isaac was describing three different strategies for text summarization in LLM. So essentially summarizing things that were in the literature. And so that was really sort of my model for trying to think through it and try to implement those strategies. And so what Isaac did was he said, we need some sample text. And so he picked the most recent State of the Union address, right, that's posted on his website. And so of course, you know, it's easy for us to just import it from his GitHub user content. And you know, a thing that Isaac did that was really just useful for us is he broke up each of the sentences into a separate line in this document. And so just a kind of an import as a list gives us each of those individual sentences. There are a few blank lines, so I just deleted the blank lines, right? So now we've got a list of sentences, right, that, you know, each entry in this list is one sentence in that State of the Union address. So the first thing that you might try, or actually, I mean, the first thing that I was curious about was whether we could just use the prompt repository to do this, right? And so, you know, we could actually try it, right? Like let's use LLM summarize, right? I'm going to live dangerously and just try to run this one live, right? So we look up the resource function and then run this, right? We're just joining the strings together and we get that error token, right? You know that we've exceeded the token length right? Now what's really interesting though is that actually there is a function on the prompt repository called Summarize URL content and that actually does do the trick, right? You know, and so I think behind the scenes there's some server side tool that is being called and that is actually. Go ahead, John.

So in the prompt repository it is actually possible you can download the definition notebooks from things in both the prompt repository and in the function repository. And so if people are ever curious about like what are the actual prompts or what are any of the pre or post processing that's going on into any of these things. It is actually possible to check. I know I actually, I haven't looked at it this week, but I know in like the very first iteration of this one thing that you would sometimes do to avoid hitting context limits is, you know, just take the certain first part of a document and that's like an easy first, you know, trick to do, right? Since a lot of the important stuff might be in the beginning. But yeah, if you're, if you're ever curious about any of these things, you can download versions from the repositories directly.

Yeah, that's a really great piece of advice. And for those of you that are looking at the notebook later on, I've included a link as well that'll take you there. But somehow it works. I guess that's cool. It also seems too easy. So I guess in the interest of trying to understand things by building them up, we could explore some of those techniques. Right? So, you know, one of the approaches is this approach called recursive summarization, right? And so the idea is we just break up the text into short chunks. Each chunk fits within the LLM window, right? And then what we do is that we. We summarize each chunk independently, right? So it's sort of like if you took a book, you. You rip it up into pages, you summarize each page, and then you concatenate the summaries of each page together and make a summary of the summaries.

Oh, okay.

So that's kind of a neat way, right? I mean, one advantage of this is that it's really easy to parallelize, right? So, you know, you can dispatch all of those independent summaries of the smaller chunks. You know, get the list of chunk summaries, right? And then it's just kind of one shot to summarize it again, which a human couldn't do.

They couldn't read each page simultaneously, but the computer can. So.

Yeah, well, you know, if they had enough time and enough coffee, I suppose so. You know, one thing, however, is, you know, I've noticed that the, you know, actually it seems like the default settings are to use GPT4 for more and more things. And so actually, you know, one modification from the community post is that I've added in this LLM evaluator option that just forces us to use GPT 3.5 Turbo instead. Right? So this will give us results that are like, a little bit more comparable to what we had before. It'll also save a few pesos, right? You know, because the GPT three and a half calls are just a lot cheaper.

Plus, you want to be using the most accessible of tools to see whether or not this method actually works for them.

Yeah, that's right. And it becomes a lot less Expensive to experiment if you do that. So, you know, just as kind of a tip for the readers. Right. You know, that's something to keep an eye on. So basically our chunking is not too hard to think about. Right. You know, we have our list of sentences, right, that we downloaded from the web and maybe just a parameter to define the chunk size. I'm doing this based on word counts, which are not exactly the same as the tokens, but it's just practically easier to think about it.

Am I correct in remembering that a token tends to be more like a part of meaning, like a suffix or a prefix, or it can be an entire word, but it's usually just whatever part means something within.

Yeah, that's right. I mean, there's a little bit of special sauce in developing a tokenization, but yes, that's the. That's the right idea. So, you know, some words like happening, you know, might be broken up into happen ending. Right. As kind of separate tokens on average. The sort of rule of thumb is that, you know, it's about like, you know, one word is like, or a token is about three quarters of a word. You know, just as kind of like a rule of thumb, there's a tool called tick token, you know, that is provided by OpenAI that will actually pre count your tokens. And in fact, there is a. There's a tool on the Wolfram function repository that will count tokens for the GPT2 model. But the tokenization in GPT2 and GPT3 and above is different. Right. So, I mean, so it kind of gives you some estimate, but it won't be exactly. Right. Yeah. So I guess there's some nuance, but. You're absolutely right, Noah. Right. A token is like. It's often a piece of a word. Although simple words like note that as Wolfram, you know, are probably, you know, just individual tokens. Yeah. So. So.

So basically, I believe you've also done some. Oh, sorry, I was just gonna. I believe you've also done some really interesting things where you've actually fitted a linear model to. To check what the. The average thing per token is, which I love when people do that sort of thing.

That. That's right. So, you know, when I was first playing around with this, I was trying to do some retrieval augmented generation. And, you know, again, I mean, that was kind of my first foray into doing this, so I didn't really know what it was, but indeed. Right. You can kind of take word counts and token counts and, you know, fit your own linear model to try to figure out what that is. This is all based on the GPT2 token counter that is on the Wolverine function repository. So, yes, I mean, it's kind of a fun thing to play around with. And, you know, this particular corpus that I was playing around with, you know, had a bunch of shorter documents. It had a few documents that were just too long. And, you know, sort of based on this, you could figure out a reasonable place to just split things and divide them into smaller chunks. Very cool. All right.

So.

Cool. So, yeah, so in this recursive summarization, I mean, essentially all we need to do is just take our sentences, right? We count the words in each of the sentences, right? We accumulate the total counts and then we basically just kind of figure out what we got to how far we are along and then use that to just divide sentences into chunks that are up to 300 words, right? So the idea here is that we don't want to just count 300 words or 3,000 words and then stop, right? Instead what we want to do is we want to say, I want a whole sentence. I never want to break in the middle of a sentence. So just take up to the amount of time that I have, right? So, so essentially what we're doing here is really just kind of like coming up with the cumulative word counts, you know, as we add in each sentence and then, you know, just breaking those up. Right. You know, doing a group by list to group the sentences together. Right. I like using a lot of these functions that use associations. But then at the end of the day, we just want the values, right? So we just kind of pull out the values in the association and then, you know, string riffle just takes those separate lists of sentences and puts them together as just a bunch of sentences concatenated, right? Just as kind of single strings.

I bluffed knowing concatenated from before and from the context. I think I understand, but what does that actually mean?

Oh, it just means push them together. So, you know, we start out with, let's say, a list of two strings and then we end up with one string that that is joined. Right? And so what string riffle does is a built in function in Wolfram is it just takes a list of strings and makes one big string with a space in between each of the pieces. There's probably a more elegant way to write this, but this is just the kind of the first thing that came to my mind as kind of a way to make individual chunks. Right.

I know I say this in these interviews a lot but if I could get students away from any idea in mathematics and science and whatever, it would be the idea, especially in mathematics, that they should be looking for the right way to do this. I hear people asking me that all the time in classrooms. It is such a toxic idea that says that there is one right way, and any deviation from it makes it.

More and more wrong.

And I just love the idea that the Wolfram language is built around, which is if it works and it doesn't cause problems, it's right.

Yeah, absolutely. You know, I think as a. As somebody that tries to teach students how to use Wolfram language, you know, I think I often, for class materials, spend a lot of time trying to come up with the simplest, the most elegant way to do things. For these blog posts, I mean, it was just me drinking coffee, trying to hack around. So, you know, it was really kind of more along the lines of whatever works. Yeah. You know, that being said, I think a style that I like and that I think works well for these types of iterative explorations is, you know, really just leaning into that. That sort of functional programming approach in the Wolfram language. Right. I mean, it's a language that is very expressive. You can do procedural programming if you want, and lots of different things. But, you know, this idea of composing functions together is really useful. You know, the other thing I guess, that I just try to do personally, because I aesthetically like it, is I like to try to do everything using width instead of using module, just because I find, well, first of all, it's a little bit faster. Usually that doesn't matter. But also the fact that, you know, it essentially assigns a constant value, and so then you can't screw it up later on in your function. So, you know, it just kind of makes the debugging, I find, just a little bit simpler. And it's often kind of a good heuristic clue, right. That, like, if you needed to define multiple things whose values change, maybe what you really want to do is define a few simple functions that you compose together instead of having some really big, long, long function. So, I mean, I guess that's just sort of like a style that I. That I find is. Is useful. No, I think it's not always the right idea, but sometimes it's useful as a guide.

Yeah, I think it's great insight there because a big part of figuring out how to do a project is often figuring out, okay, what functions am I going to need to define in order to make the, you know, look clean and. And I can understand what it is that I'm doing. So that. That's great advice for people.

Yeah, go ahead.

If, if only for the reason that once you've done it like this, if something is going wrong, you can test the individual parts of your system and figure it out. If it's all just one cell, that is an incredibly long, incredibly complex, possibly brilliant combination of code. And then it doesn't work. It's somewhere in there.

What. Yeah. What are you going to do? Right. And so, you know, also, as kind of an approach, Right. You know, not shown here because I've deleted it, is often in the process of trying to debug. I mean, you just use that sort of interactive nature of the notebook. Right. So you've got some test example. Right. You can make sure that the little function does the thing that you want it to do. Just eyeball it. Right. You don't have to write any formal tests. Right. But that's kind of a good way to iteratively develop those small pieces and then put them together. Right. And I always like to tell students, like, just solve a small problem first. Like, just solve one case, and then if you've got to apply it to a bunch of cases, then just apply the function that works on one case to all the cases, like using a map or something like that. Right. So, yeah, all good ideas. I mean, here's. Here's how we. Go ahead. Oh, yeah, sorry, Jason, go ahead.

I was just going to say it shows the parallels between writing and coding and that just like in when you deal with writer's block, you sometimes just want to free write. And what you've been describing is free coding. You're just getting things down, getting it out, not worrying about how it looks. If you are trying to teach stuff, of course, you may have a metric of best by being the shortest amount of lines or, you know, well functioning code or. Or if you're doing production stuff that you need to, you know, have tens of friends, you know, use, then you might need to have to worry about speed.

But.

But it's important just to jump in and get some stuff done. And Wolfram language really helps with that. But it's just impressive how science and coding and just writing could be so similar, even though so different.

Yeah, that is a really great, great analogy. So, yeah, I mean, just to kind of show how this works. Right. I mean, so.

And for anyone who's joining us right now, the summarization method that we've been talking about so far is to summarize parts of a long document and then summarize the group of summaries.

Absolutely. Right. So, so jumping in, right? I mean, the first thing we have to do is just define. Divide things into chunks. So we define some function to do it. Right. This is a function that satisfies us. Right. And then the actual summarization is actually, you know, pretty straightforward. Right. I mean, you get the whole corpus as a list of sentences, Right? The whole. In this case, we're summarizing a State of the Union address, right. We use that chunking function that we defined to break it up. Right. And so now we've got lists of chunks of sentences, right, that are, that are not too long. And then, you know, we just apply this summarization function to that list of chunks. Right. And you know, what I'm doing here is actually applying the LLM function using parallel map, right? Because we want to apply the function to each chunk. It takes a few seconds to send the data to OpenAI and get the response back. And so we're just taking advantage of the fact that we've got a few parallel processes to kind of like run a few of those at the same time. It's not really necessary. You could do it as a sequential map.

But although I do recommend doing it in the way that Josh has just described, because you'll be waiting quite a bit longer if you try to do them in sequence.

Yeah. I mean, four times as long on my home license.

And that's especially a problem if you were trying to summarize something like a book as opposed to a State of the Union address.

Yeah, that's right. And I guess, you know, if I could put in a wish list for the elves at Wolfram, you know, it is certainly possible in the OpenAI API to run parallel calls. LLM synthesize and the sort of LLM resource function calls don't allow that in the current version or, you know, it's not kind of the way. The way that it's implemented, you know, but it's, you know, something that, that could be there in future issues. This is kind of a way around that, you know, just taking advantage of, you know, running these things in, in. In parallel. Yeah. So then we get back the results, right? We've got the summaries of, of each of these chunks, and then we just put all those summaries together and then divide them up into sentences. Right. And now here's the part that's recursive. Right. So we're actually going to run this process of summarizing chunks over our list of sentences as long as we exceed the word count, right? Because we wanted to get a summary that was below a total word count. So basically, this kind of first stage or running this process once gives us a shorter document, right? We've summarized each of the individual pieces, but the sum of the summaries, right. The aggregate summaries may still be too long. And so essentially, we just use the kind of nest while function in the Wolfram language to just repeat this process until we're below the desired word count. And then when we're done, we just put them back together, Right? So that's the recursive part of this recursive summarization method. You know, again, you know, we have similar type of pattern, right? We just define a function that, you know, returns true or false if we exceed a, you know, the word count limit, right? And, you know, that that's simple, right? We just take the sentences, we put them together, we count the words, we see if it's greater than. And, you know, that becomes our guard statement in this nest while. Yeah, so that's one strategy. And, you know, rather than just run this, you know, I have kind of the result here. You know, we wanted it to be below a certain number of words, and, you know, we ended up with something that was at like, you know, 210 words and pretty much summarizes last year's State of the Union address, you know, as I recall it be. So that's one way to do it, right. Another strategy that you can use for summarization is what's called the refining method. And actually, this is a lot like the strategy that you had suggested, Noah, back at the beginning, when we're introducing this, the idea is that we're just going to march through our text and summarize the first chunk, like the first page of the book. We'll write a summary of that. We then say, summarize that summary plus the second page of the book, right? And now we'll get out a new summary. You know, that is the summary of the first summary plus the summary of the second page, right? And then, you know, we just keep marching on until we're all done with the. The total amount. So that's neat, right? It's kind of easier to implement. So maybe it's a thing that we would do first. But, you know, a disadvantage is that it's a serial process, right? So we can't really take advantage of that parallelism, right? Like, we've just got to kind of march through chunk by chunk and, you know, people that study LLMs have also tried to kind of see what happens. And maybe there are like pros and cons associated with this type of method, right? Like you might lose some of the information in those initial, initial parts. But again, it's another popular method of doing it. Something that's totally reasonable. And you know, as you can see, it is, it is pretty simple to implement, right? So less code involved. Yeah, right. So you know what we're going to do, right? I mean, this is, you know, how we'll apply it, right? We have a list of the sentences, right? And our desired chunk length, right. You know, how big is each page? We could just use that same chunking function that we wrote up above, right? So that's good, right? We've got a list of the chunks that we want to summarize and then what we're going to do is we're going to fold this function over those chunks, right? So those of you that maybe aren't familiar with this sort of functional programming idea, right, what fold does in the Wolfram language is that it applies a function, right? In this case this join summary function to an input value, right? And then we've got kind of a new result that's present, right? We apply the function then to those two arguments and those two arguments and so on, right? And in fact, actually I think the third list explanation is a little bit clearer, right? So, you know, our first iteration just gives us the value A, right? You know, the first thing in our list, right? The second value gives us F of A and ab, right? Our next iteration gives us F of F of A, B and C and so on and so forth, right? So it's like folding it over each of those inputs, incorporating that next thing. And that's exactly the type of thing that we wanted to do, right, in this process, actually.

I love that. This is probably the most down to earth example of using folds that I've ever seen, you know, even though, yes, it's using an LLM at some part of the backend process, which is, you know, okay, fine, that's, that's some fancy thing, but like it just is an example of what fold can do. I love it.

Yeah. Oh, great. Yeah. I mean, hey, who knows, maybe it'll be in the next fold documentation, right? So you know, the, the function that we're folding over, right, Is this join summary, right? And you know, all we're going to do is we just have the previous summary as the first input and the current chunk, right, of new material that we want Right. As the second argument. Right. You know, again, we just define a little constant. Right. Input. Right. Which is just take the previous summary and put it together with the chunk, right. The new chunk that we want it to do, and then we just send it to our LLM summarize. Right? That's it. Right. And then this will return that new summary. Right. Of that pair. And then we just use fold to repeat over that until we're out of chunks. And then when that's done, it just returns what the final value is. Right. So yeah, it's really neat. You know, we can see how well it does. It ends up being, you know, a bit more concise. But, you know, I think we could also play around with that a little bit interesting. Maybe setting some of the arguments about the summarization chunk sizes.

And that makes a bit of sense too because in this situation, each summary of the original material has been summarized multiple times by the time it gets down to the end of it.

Yeah, that's right. And so, so, well, at least the.

The ones from the beginning of the document have.

Yeah, that's that. That's right. So, you know, a potential pathology in this type of strategy is that you emphasize information that comes at the end. And you know, unless there's something that really kind of catches the attention of the summarization function, you might lose some of those details that were mentioned earlier.

As a homework assignment for people who are watching. A really fun thing you could do to iterate on this is to create a string template prompt, maybe even add a few, a couple of. Few shot examples of what you're supposed to. What it looks like to do a good job of taking a summary and appending a little bit, both in the case where it is highly relevant or in the case where maybe it's not relevant to what's, what's. And see for yourself, you know, what, what the sort of differences are in ways that you could iterate on this sort of strategy.

Yeah, that's a really great idea. You know, if someone wants to implement that, I'd certainly be interested to take a look. Right. And you know, there might be something to just kind of tack on as a, as an add on to this Wolfram community post, because I think, you know, that's a really.

Which anyone can comment on.

So anyone can comment on that. Absolutely. I guess we could talk about another strategy, right. This one's kind of neat, right? I mean, the idea is that you actually combine summarization with topic modeling. Right. And so the idea is that we're going to basically take this speech, we're going to divide it up into groups of 10 sentences each, right? And we're going to overlap them by two sentences each for continuity, right? So it's kind of like an overlapped list of things, right? So we take sentence 1 through 10, then we take sentence 8 through 18, we take sentence 16 through 26 and so on, and divide it up in that way. We're going to summarize each chunk and we can do that just straight out of the run, right? Because they're only 10 sentences each. But then we're going to do something else, right? We're going to compute an embedding vector, right? That is like an internal representation of what that text means, right? It's a 768 dimensional vector, right? Oh, my goodness. That describes the internal state of the GPT model, right? And then what we'll do is. Now those are just vectors and so what we'll do is just cluster those embedding vectors, right? There are lots of different ways to do this, right? In the kind of example that I read about that uses a particular community detection algorithm, modularity optimization, you can do that using the Find Graph Communities function in Wolfram. But, you know, you could use anything that you want to kind of summarize and cluster the entries, right? At this point, each of the individual chunks is just a vector in this 768dimensional space. And so anything you could do to cluster vectors is something that you can use. All right, so here we go. I'm just going to demonstrate this for like one round, right? I guess, showing this idea of like, just build a function that does it for one thing, and then put it together, right? So we want it to chunk by sentences instead of word count. So we need a new chunking function here, right? We take our list of sentences, right? We have our chunk size and our overlap. And actually, you know, you can do this just using the partition function that is built into Wolfram, right? So we say partition the sentences, they should be up to a particular chunk size, right? Like we might not have a complete multiple at the end. So up to just allows us for the last few to kind of be below our chunk size. And then, you know, we want it to be offset by the chunk size minus the overlap, right? So that just kind of marches through and gives us that sort of 1 through 10, 8 through 18, 16 through 26 and so on, right? That's it. So that splits it up and then we just use a string riffle to put those sentences together into individual strings, right? So that's one little thing that we have to do. Then the first step is we generate the summaries for each chunk, right? And we've already seen how to do that, right? So we, we split them up into the chunks. We just use parallel map to. To then, you know, generate the summaries for each of those 10 sentence chunks. And, and here I just saved this because I didn't want to redo it. You know, it takes, takes a minute or two to, to do. So let me just retrieve the ones that I had from before so we can play around with them. The next thing that we need to do is generate the embeddings, right? And so you can do this with the kind of, the default Wolfram language support. You have to use the, the OpenAI service connection. So there's not like a function like LLM synthesize for, for embeddings. It's kind of baked into the, the OpenAI service connection.

You can access that either through doing the Christopher Wolfe from OpenAI Link, as Josh is showing here, or also, as Josh was saying, the Service Connect and then Service Execute.

I have to say, back when I was trying to figure this out in August, I hadn't figured out the Service execute, OpenAI service execute stuff. So I just use the Christopher Wolfram's OpenAI link, and that defines a function OpenAI embedding that will make the calls and compute the embeddings. Right. I think now that I've discovered the Service Execute, it is actually a bit more powerful, right, because it actually allows you to request multiple embeddings simultaneously in parallel, although it's limited to about 20, you know, anyway, it's kind of small detail, but, you know, these are. Oh, sorry to do it. Yeah. Oh, go ahead.

Sorry. I was just going to say, also stay tuned for, you know, things coming up with being able to register services in upcoming version because, you know, we're in a sort of phase of, you know, technology going on in the world where there's interesting new things that you might want to service connect to all the time. And so, you know, you might be interested in some mechanism of, oh, there's a service, you know, there isn't like an official function for it yet. You know, maybe I don't want to worry about writing HTTP requests because, you know, you can do that, but that can be, you know, not necessarily the most elegant way. And so maybe you want to create some service request that isn't one of the predefined things so stay tuned everyone, for things going on with service connections that should, should help in some of the things that Josh was figuring out, you know, a while ago that, you know, I agree. There's actually some times where I do resort to writing either the service connect or the HTTP request, depending on some specific little corner case I might want to do.

Yeah, yeah, no, that's really great to know, John. You know, I think that'll be super powerful, right? Because, you know, so many of these really powerful services provide rest APIs, right. And so having a kind of a really convenient way of working with that is going to be super empowering for people. So anyway, you know, at the end of the day, I mean, but if you just use OpenAI link, right, you could kind of generate all your embeddings. Doesn't take too long. It might cost you about a nickel to run this experiment. Experiment. And you know what you end up doing is that you end up kind of getting a list of these 768 dimensional vectors, right. For each of these 10 sentence chunks. Actually, just to visualize this, I use distance matrix and we're just using the cosine distance, right? So that's kind of like, that's just the dot product of two vectors. That's kind of like the standard way that people try to look for similarities in these embedding vectors out in the community. Right. And so we can see that like, yeah, every, every chunk is exactly the same as itself. So, you know, the diagonals have the highest values. Yeah. And then it falls off. But you know, what's neat is that you can kind of see that there are like thematic, you know, semantic relationships between sentences that are not next to one another. Right. So the kind of lighter areas are areas that are kind of like semantically more related to one another. And so what we're going to do in this approach is essentially just cluster them, right. So we could just use, you know, the Find Clusters function built in, built into Wolfram. Right. You know, we're actually going to do a little trick here, right? So we're clustering by the embeddings, but the value that we want are actually the summaries that are close to one another. So if you just use this embeddings goes to summaries, Find clusters will return the word chunks that are similar, but after they've been clustered by the vector similarity, right. As measured by the cosine distance. And so then that gives us things that are, that are related to one another, and we just put those strings Together, right. Using. Using string, riffle.

Oh, okay. So the summary will be talking about similar things at the same, in the same area, as opposed to talking topic A, then topic B, then topic A again in some weird sort of fragmented way.

Yeah, that's exactly right. So like, for example, like in these chunks, you know, actually, rather than like putting the text together, which would be too much, right? Would you say like. Yeah, chunks, you know, 1, 2 and 23 were all kind of thematically related to one another, right? Like the words had a semantic relationship to one another as defined by the GPT model. Right. So those should all go together, right? You know, chunks 3 and 42 had some relationship to one another, right? You know, chunks 4, 5, 6, 8, 9, 11, 13, etc. Right. You know, were related to another. So what Fine Clusters is doing is essentially saying, yeah, these are these things where like the, those vectors are all kind of like near one another in this, the 768 dimensions, dimensional space. And so, you know, we should kind of put those, those together, right? And you know, I, I think I counted also. Like, you know, if we, if we look at these chunks, like there's a pretty big set of passages that, that all are supposed to go together, right? We might be worried that that's, that's too many words. But it turns out it's not, it's not too much, right? It's only like 1300 words. So, you know, if we were writing production code, we might want to think about that edge case. But, you know, just for the purpose of hacking, it was good enough. So, you know, essentially all we have to do, right, Having kind of demonstrated that, that like each of the pieces, right. What we're going to do is just define a function, right? We get a list of sentences, right? In, in our document, right? We generate the summaries, as we demonstrated up above, right? By breaking them up into these overlapping blocks of 10, we compute the embeddings as we, we demonstrated above in this case, using the Christopher Wolfram package, we find the clusters, right, as demonstrated above, and then we return a list of the kind of, the new chunks of things that should be all together, right? So that's it. That's one iteration through this process. And so then what we'll do is we'll then just again use this sort of functional programming model, right, where we're going to run this process of summarizing the clusters over. We start with all the sentences, but in the next iteration we've got the summaries that have been grouped together and summarized together in appropriate chunks. And we just repeat this until we no longer exceed the word count. Right? So that's it. Right. And then when we're all done, we've got this list of kind of summaries that we put together. We just stitch them together with string riffle and so, oops, we can just see what that ends up giving us, right? Oh, hang on a second. There we go. So, I mean, this gives us a summary and actually it keeps a lot more of the details together in this summary, right? Because it's essentially said, yeah, like these were things that were disconnected in time in the speech. Right? But they were themes that should have gone together. And you know, when I put them together then I realized, like, that's actually an important theme in the. In the speech. Right. It's something. I think I should keep.

This. This is fascinating. I want to say first of all that I can't believe this is what you do when you are idly curious in your free time about something. This is amazing. I do want to ask though, do we have some more examples that are a little more chemistry specific, maybe in terms of either stuff that you're teaching or stuff that you're doing in research or anything like that?

Sure. You know, and you know, I guess I could. One thing that I think is. Well, I'll tell you about one thing that is related to research that I just think is really neat, right? Maybe going a little bit off script, right? So there's an idea of tool use in these large language models, right? So the idea is that the large language models don't know how to do precise calculations, right? But they can call other tools to do those calculations tools that are defined by computer programs. And those of you that have used WolframAlpha as a plugin within ChatGPT have seen how this works, right? ChatGPT calls WolframAlpha that then does some calculation. And so I was really inspired by a research paper that I saw called Chemcro. And these were some chemists that said, hey, I've got an LLM. Can I add in some chemistry specific tools that the LLM can use to perform calculations that it needs? So when I first read this paper, it sounded really hard, but it turns out it's actually super easy to implement this in the Wolfram language. So in fact, I'll show you this in like three minutes, right? So you know, just as an example, right? Like here's some molecule, right? And you know this is all default Wolfram language code, right? That takes the name of a molecule and computes a standard identifier known as a Smile String or computes the molar mass, right? Something you might do in high school chemistry. You know, ChatGPT doesn't really know how to handle complicated molecules like this. And so if you ask it what's the smile string? It just gives you the wrong molecule, right? So that's weird, right? And so basically what you can do in the Wolfram language that is really powerful is you can just define a tool for your LLM, right? It's just LLM tool, right? The first argument is the tool name, right? So I'm just going to call it Name to Smiles, right? This the second argument, or I guess the second argument is the second entry in a list, right? So this is actually a description of what that tool does that we're going to tell our GPT model, right? So this, this is what I do, right? This is my name and this is what I do, right? Our next argument is the actually the input parameter variable, right? I'm just going to call it Name because I couldn't think of anything better. And then the last argument is the function that you actually want to perform, right? And this is something that the Wolfram language actually knows how to do, right? And so then you can use this tool in LLM synthesize really easily, right? You just provide it as an input to your LLM evaluator option, right? And so now when we ask this, oh, I guess I would help if I define this function. I'm living dangerously here. So now when we define this, you know what the LLM is going to do is it's going to say, hey, I've got this tool that I know converts a given chemical name to its corresponding Smiles representation. I need to use that to solve this problem. Okay, I'm going to go and ask the Wolfram language implementation running on my laptop how to do it, and then I'll incorporate that result and return the value. Right? And so what's really powerful is that actually out of the box. You know, models like GPT4 know how to put multiple tools together, right? So, you know, again, I'm just going to define two more tools that were in the Chemcro paper. They're like one liners in the Wolfram language, right? Like, here's a tool that calculates the molecular weight given one of these, these standard identifiers, right? Here's a tool that uses the resource function to pull something up. And so without this tool, GPT runs through some calculation. All the numbers are Wrong. It fails the Gen Chem final exam. Right, but with the tool we can ask the same question and we could tell the model, use the tools that are available. Even if you don't, it's smart enough to use the tools that are available and it stitches together those intermediate results and gives you the right answer. Right. By actually doing a calculation using the Wolfram kernel running on your own computer. So it's kind of like having a little back and forth interchange with the program that you written.

Thank you so much for this example. I think it's quite clear to people who follow using LLMs for research applications or for more than, let's say, writing an essay for you, that the ability for them to call functions and use tools and tap in when they want to have an actual computation done, and they do so in a fairly intelligent way, especially with the right prompting, is really one of the most exciting things about them.

It's really cool. And I have to say, first of all, the chemcro work is really neat and they have a lot of fancier tools that aren't just one liners that do things like try to figure out a chain of chemical reactions that solve a problem and so on. These were just a few really kind of like, you know, basic things that they did, but that are really, really kind of powerful. Right. And yeah, and I agree, I mean, when I first kind of learned about tool use, it seemed like something that was really complicated. And you know, and I hope this example is maybe one for the listeners on, you know, how they can think about, you know, taking those pieces of the calculation that are easy to do and easy to define a program for and you know, giving that as a capability to an LLM to, to, to work with, you know, where the input to the LLM might be a little bit messier and kind of uncertain. And you know, there are applications, of course, far beyond chemistry. But you know, this is one that I thought is really kind of neat.

That is amazing. And another one of the things that I know that you're very passionate about as well is chemical education. And so we'd love to also get into this topic with you today.

Sure. Well, I mean, that's another community post, right? And this was actually a paper that we put out just this past fall in the Journal of Chemical Education and after testing it on a group of first year general chemistry students who volunteered, the graphical table of content image was made by Generative AI and we put that in the acknowledgments, by the way, try to slip one Over. And this was actually work with my good friend and collaborator Alex Norquist at Haverford College and then a bunch of undergraduate students, Gabe Jones Thompson Kachinghe and Thomas Egg, who both Gabe and Kaching really kind of worked on developing the experimental side of the laboratory. And Tom also worked on the kind of computational materials and teaching materials. And the idea here is really kind of like to bring some of the things that I like in my research laboratory automation, using robots to perform chemical reactions and data science methods and to try to see if we could bring that to the kind of first year chemistry classroom, right. You know, we don't want to, you know, get rid of general chemistry, right. We're always going to have general chemistry class. But you know, the idea is, can we take some of the labs that we have in a general chemistry class and that other people have in labs in general chemistry class and re envision them, right. As kind of involving using robots to perform reactions and using some computational tools, right. In Wolfram, for example, to do a data analysis. And in a way they're really synergistic because with the lab automation, the students are now performing 100 experiments in 30 minutes. And so now they've got to make sense of that data. They can't do it in Microsoft Excel or just by hand with a calculator like they're used to. They really need kind of a computational environment to be able to make this work. So I mean, basically the thing that we did was, and these are all kind of like materials that are in a community post is we wanted to teach these ideas, introduce some programming in Mathematica in kind of a notebook environment. It turns out that like half of our students have already used Mathematica by the time they come into this lab, they've used it in their math classes, right? So that's kind of a. It makes it a little bit of a softer start. And you know, what we have them do is kind of work through. There's a pre lab notebook that they work through on their own, right. It includes basic things like how do you use the Mathematica interface? And like what are variables and functions and things, right. And then in that pre lab exercise, they kind of work through some interactive pieces, right. You know, that are again just written in the Wolfram language to understand some of the basic physical ideas, right. So we have this idea of, you know, sort of understanding mixtures in a geometric form. Right. And kind of understanding, like if I mix solutions A and B together and they have different species present, that's Like a point in this concentration space that we call it. So they've got some interactive exercises to work on that and to kind of think about and think through. You know, we then kind of introduce this idea of, like, well, how would you write a function that could maybe convert concentration to volumes? Right. You know, because when you go into the lab or maybe for this audience, when you're making a martini, you know, you don't specify what the concentration is of alcohol in your final martini. You specify, well, I have three parts of gin and one part of vermouth. Right. So you do everything in terms of volume fractions in the lab, but you want to think about it in terms of concentration.

Nobody's pouring by moles.

Yeah, that's right. That's exactly right. So we kind of walk them through, just kind of thinking about this and some simple ways. There are also some hints like, hey, maybe you're taking linear algebra. Doesn't this look a lot like linear algebra? Right. You know, as kind of a wink to students that maybe have a little bit more. And, you know, then the thing that we do is, you know, we can kind of use this to kind of walk through the idea of, like, what are the mixtures that are possible, Right. Like, if I mix two things, I can never get something that is more concentrated than what I started out with. Right. And so mathematically, what that defines is. Is a convex hull, unless, I suppose.

It causes some sort of huge evaporation thing to happen.

Yeah, I mean, that's. That. That would be sort of an edge case, right? Yeah. So we'll. We'll leave that. We'll leave that edge case out, right? But, yeah, so, so, so basically, we kind of walk them through this idea and, and, you know, then a possible experiment is just a point within this region, right? So, you know, there's actually something really nice about thinking about mixtures in this, this way as these convex combinations, right? The convex hull of your stock solutions defines the possible spaces of mixtures. And then you just sample random points within that. That, that. That space. Right. It's really kind of like kind of a neat and really kind of powerful way of thinking about. About mixtures. Right? But this allows students to kind of like generate randomly generated experiments, right? You know, in. In this space, and then we. We walk them through, you know, kind of thinking about, like, how do you think about chemical equilibrium in this concentration space? Right? Well, by the way, you just do a little algebra, right? You take your equilibrium expression and divide both sides by. By x. Right? And, you know, this then kind of allows us to kind of think about this, right? The. The gray triangle are possible mixtures, right. That could be made blue or maybe some. Some random experiments. And this red line indicates the equilibrium. Right. And so everything above this line is something where the reaction is going to produce a solid product. In this case, right. We're precipitating lead iodide. Everything below the line is. Is subsaturated. Right. So it's not going to produce a solid product. Right.

So in essence, this is a phase diagram.

Yeah, that. Yeah, that's right. It's sort of a phase diagram in units of concentration instead of in concentration and temperature.

Sorry if you said this already, but what does the. What do the axes represent?

Oh, so the axes here are just the concentration of chemical species X and chemical species. Yes. So in the experiment that the students are actually going to do, it'll be lead and iodide, right? And so they'll precipitate out a lead iodide solid, right? Yeah, yeah, that's it. Right. So I mean, the thing that we want students to play around with is this idea that, like, let's say you do an experiment, right? You know, you've generated some random points, right. In concentrations of lead and concentrations of iodine. Right? Now we're using real data. You know, you observe that you formed a precipitate for the red points, right? Like you're making a yellow lead iodide crystal on the bottom of your. Your container. Right. And in the black points, no reaction has occurred. Right. It's just a clear solution. Right. And so, you know, the thing that you can think about doing then is trying to, like, you've got this equilibrium equation, and by changing your equilibrium constant. The goal is to try to find a value of that equilibrium constant that perfectly divides the two outcomes, because that's really the boundary. Anything above that equilibrium curve is super saturated and will form a product, and everything below is not saturated enough to form a product. So, you know, this is a way of maybe kind of turning around the problem that you would typically do in a Gen Chem lab where you do some, like, tedious set of titrations, right. And try to, like, just march along this red curve. Instead, we just say to the students, perform a bunch of reactions, right? Observe these discrete outcomes. Like there's a yellow crystal at the bottom. That's a red point or not. Right. And then I can find where the equilibrium curve is by just fitting a dividing surface, right. You know, finding the kind of the value that best separates yeah. Things that precipitate or not, I've said.

Off air, but I'll say it again, that, you know, had my chemistry education been this interactive and, you know, this presentation of things like equilibrium constants, I might have, you know, been more interested in going into chemistry instead of other sciences. This is fantastic.

If only we had a time machine as a program.

There's still time for you to switch.

Yeah, that's true.

You can see the light.

Lifelong learning is definitely an important thing for all to keep in mind.

Absolutely. So, you know, of course, you know, we're just. I'm just kind of running through this pretty, pretty quickly in the interest of time. You know, there's kind of more to describe. You know, it turns out that if you take a logarithmic, then it just becomes like kind of a linear problem. And then that actually is exactly a logistic regression problem, right? So we turn this problem of finding an equilibrium constant into a logistic regression problem. And then, you know, just use Wolfram language to fit that logistic regression problem. Right. You know, where we're trying to find this dividing surface. So we kind of walk students through doing that and then we say, all right, great. Well, you've kind of worked through this fake example. Now it's your turn. You know, we want you to generate some experiments, right? And you know, we're going to have. We're going to run them on a robot, right, in the lab. In fact, you're going to run them in the lab on the robot yourself. So basically what we do is we have the students generate, you know, a list of experiments, right? They generate, you know, kind of a 96 experiments each. We then have them do 24 of those experiments by hand. It's really tedious and annoying, right? But they actually learn some useful skills, right, of using a micropipettor. And then we bring in students in pairs to, to work on the robot, right? And so they take these machine instructions that they've generated. We show them how to use this, this liquid dispensing robot, and then they fire it off and they run the reactions. And then, you know, the reaction actually takes about like, you know, 24 hours to occur, right? It's a little bit slow. So they set everything up and then, you know, just put the, the, the setup that they, they made aside. And then we have the TAs just go through and take a digital photo and dispose of everything. There is some lead that is involved. So, you know, it was just nicer to have the TAs do it. But yeah. And then, you know, this is the sort of the pre lab. You know, there's also a post lab notebook that we have students run through that really says like, okay, great, start with importing your data. And now let's work with that data. Right? Let's analyze your results. Essentially, it's like items 7, 8 and 9 in this list, right? They perform the experiments, they read the results in, they do some visualization, and then they find the equilibrium constants. And then it turns out that the values that they get are going to be different from the ones that they did by hand. And so we have them kind of analyze that and think about sources of error. We also have them aggregate all their experiments. So we had 14 students running this. So there were seven batches of experiments. Each pair of students ran about 100 reactions. So they have 700, 800 reactions to analyze in aggregate. And so now they can really start to see statistical properties emerging and get a sense of what are the variations. And as soon as they've got 100 experiments to analyze, there's no way that they're going to do this by hand. Right. And you know, fitting a logistic function is not something that they can do in Excel either. So, you know, it really kind of motivates using some of these tools. You know, logistic regression is kind of like the first thing that you learn in a machine learning class, right? Maybe after linear regression, kind of related. So, you know, the goal is to kind of expose students to this, this idea and just plant a seed that like automation is relevant to chemistry, data science is relevant to chemistry. And what was really neat, you know, you can kind of read more about it in this, this, this paper that we wrote. I mean, we surveyed students at the beginning of the, the class. You know, all of them knew that automation and robots existed. Some of them had even like built robots in high school. All of them knew that machine learning and data science were things that existed. But only 1 out of 14 students knew that you could apply those ideas to chemistry after the fact. They all knew that you could do it, and most of them really liked the idea. So hopefully we're kind of training a generation of students. Some of them will be chemists, some of them will be biologists, some of them will be physicians. Right. You know, who, who know that like data science and automation are things that are, that are relevant to scientific problems. Right. And, and they've even gotten a chance to, to get their hands dirty in the lab and get their hands dirty on the keyboard programming some of These things and, and building them up, they'll.

Be ready to use the laboratory of tomorrow, which is actually arguably driven by a Mathematica notebook. If you're familiar with the Emerald Cloud.

Lab, I very much.

Where you can, you can just write. Write your experiments in a Wolfram notebook and run it.

So that's really cool. It's really exciting technology. And you know, and I think, you know, as that becomes more accessible and affordable, you know, it's really fun to think about how do we integrate that into education. And I know our friends at Carnegie Mellon University are kind of pushing that forward. You know, this lab, even though we didn't use the Emerald Cloud lab to perform it, we had the students actually go in and do the experiments themselves, you know, is absolutely something that would transfer really smoothly into that sort of cloud lab type environment.

This is beautiful. I've got. I'm sorry, did you have more you wanted to say there about.

No, I think I could just stop right there. Okay.

I've got a question from the community coming in asking. And this might have been the regression that you were talking about, but I do want to double check the division between saturated and unsaturated. Is. Is that just something that they would try and guess by eyeballing it or was there actually an algorithmic approach to finding the division there?

Yeah, you know, sorry to run through that.

So, I mean, I'm glad you're getting so much in on this time, but yeah, so.

So, you know, what we wanted to do was like, we have them actually try to just kind of like use the slider bar and the interactives to get a feel for it. But in the end, yeah, actually we have them do a logistic regression and.

That'S the regression you were talking about.

Using logit model fit. So they actually do it. They first do it kind of like in the loosey goosey interactive way to get a feel. But then we say, you really don't want to do this. Right. And by the way, like, how are you going to make a disciplined choice? Like, there's going to be experimental error. You're not going to find a perfect dividing surface. You could eyeball it and make a guess. Or, you know, we have kind of a well defined algorithm that will do it. So in the end, they use logic model fit to find the best parameters and do the calculations on those parameters and, you know, come up with the final result. You know, we also show them how to add some extra constraints. Like, you know, there are different variations of it that we walk them through but you know, the goal was like give, give them something to play with interactively first just to get the intuition, right. The kind of the fingertip feel and then show them how to do it interactively. And the same thing with the concentration space. Right. Like we have these kind of interactive exercises just to get a feel of how those mixtures work. And then we say, well actually you just want to do it as an equation solving thing. Here's how you solve it as an equation. Right. So the idea is also kind of provide kind of like a multimodal learning experience, right. If you like fiddling around with things, fiddle around with things, and if you like writing a program, write a program. Ultimately we want everybody to do both so they can make those connections for themselves.

This ties exactly into the second thing that I was going to ask you about, which is that I love how there's such a combination in these examples of you saying I have some tool which does a particular thing very well. How can I change what I'm doing into that particular thing so it will be successful. And other examples where you have things like fold, where you say I want to do this particular thing. What is a built in function that will do exactly that sort of thing for me? And so I wanted to ask you in either research or teaching or whatever, which of those experiences do you most often find yourself having when you're working with these tools?

Oh gee, you know, I think, you know, I've been programming since I was 5 years old and that was a long time ago. Not in Wolford language, in basic. Right, but you know, but, but that was a long time ago. Right. And you know, I've programmed all sorts of different languages, supercomputers, 25 cent, microcontrollers. So I mean, I guess I'm at a certain stage where I generally have an idea of hey, this is something that I want to do as a functional fold and I need a function that I can fold over. Let me implement that function. Right. So I mean, I guess like for myself as just a kind of an implementer of things, I, I've usually got a pretty good mental model of how to break up the problems and I just start at the smallest layer and, and build upwards. Right.

We had an audience because I've been.

Around the block a few times. Right, sure. As to your question of like kind of finding things that are built into the language that, that work, I mean, I think my old practice was like just reading the documentation and kind of searching the documentation, but I think now actually using the chat notebooks is really powerful, Right. Because you could just open up a chat notebook session and say, hey, this is the thing that I want to do. And it's often not the way that I would implement it, but, you know, it tells you what are good functions that you. That you might use.

I'm glad you mentioned that because we, we just had one audience question related to that, which is a. How do you help students? You know, if there's any kind of a learning curve in getting into the coding initially. So that, that's a great answer to that question. Another audience question that we just had has to do with, you know, so is this a general chemistry course? Because we mentioned students who are thinking about going into biology or physicians or things like this. But it's also important, as you well know, for, you know, students who, who knows what they're interested in doing. Yeah. So the question was, you know, how did those students like this sort of approach? Did that connect as well as I, I would expect just from like, education three level? It did. But what were your, your actual experiences with students that. That might not be interested in going into chemistry, but we're still doing this experience.

Sure. Right. I mean, I think that, that that's, you know, so that's. First of all, that's a really great question. This is kind of a new thing. Right. So, you know, in the past, one of the things that I had worked on. Right. Was trying to incorporate more of this sort of programming and machine learning topics into our advanced chemistry courses. Right. Like junior level chemistry courses. Right. I even wrote a book on essentially how to implement all these basic models of physical chemistry in the Wolfram language. Right. Christmas is coming, in case anyone's looking for a gift idea. But, you know, this idea of trying to work things into the general chemistry curriculum is brand new. Right. I mean, we just tried it this past spring. Right. So. So this is really brand new. All of those students are like sophomores now. So, you know, so we don't know how they've gone on in their career trajectory. But I think something that's really exciting about working these ideas early into the curriculum is that you do get the broadest cross section of students. Right. The students that take general chemistry, you know, might want to go into the sciences, any one of the sciences. They might want to go to medical school. They might decide, hey, this is not for me. And, you know, but, but yeah, I, I think the thing that we can do is kind of introduce them to some tool, some ways of thinking about the world Computational thinking. Right. And that's something that is, is really generalizable and can take you in lots of different directions. Yeah, you know, I, I also teach a course for non science majors on drug discovery and you know, computational drug discovery. It's also a writing intensive course and you know, it's a chemistry course. So we have to have labs, they're all computational labs. And so we do some, you know, basic Wolfram language, cheminformatics things. We also do some, some protein docking, drug docking calculations. And you know, that's an audience of students that are like not at all science majors. Right. This is a class that doesn't satisfy your requirements as a science major. Right. They're often philosophy majors or English majors or whatnot. And you know, those, those students really get a lot out of the sort of computational thinking type skills because, you know, it's something that at the time they say, gee, this is really kind of hard, it's really different. And then later on in their coursework they come up with something, a problem that they say, oh gee, if only I knew how to calculate a net present value. Oh wait, wait, we did that in Shrier's class. Let me dig back in and try to do this. Or gee, I've got this data set of psychometric things that I need to plot and fit. If only I knew. Oh wait, I did that before. So yeah, so I think in a lot of cases, especially as we put these things early on in the curriculum, we're really planting seeds and students can take that in lots of different directions. I've got one student who actually I think I'm going to meet with in six minutes who was a student in that class and as a, maybe partially as a result of that, you know, decided that, you know, he really wanted to get into mathematics and data science and is now playing around with LLMs for different things. And so, yeah, so you never know. Kind of like the way that students are going to take things. But I think if we give them some more tools in their toolbox, it only helps them. Right.

I love this because a question that is so often seen as confrontational in education is when are we ever going to use this? You know, is looking at something that you're being taught and responding by saying, I have no idea how this is going to connect to anything outside of this class. And it sounds like the answer to that in this particular case is absolutely anywhere and wherever you want to. Because all of these things are things that could be applied so much more broadly than just the particular problem they're being used on.

Yeah. And maybe that's an exciting opportunity for, you know, especially with these sort of LLMs is, you know, they give us sort of a bridge between the messy, unstructured world of natural language and computable data. Right. Things that we can perform computations on. Right. You know, if I've just got the State of the Union address, it's not clear how I say that, like particular parts of that are related to one another. Right. But what, you know, the LLM lets me do is it lets me turn that into a vector and then I can see what, what parts are related. And you know, as applied mathematicians and friends of applied mathematicians, you know, we know a bag of tricks. But that bag of tricks now becomes relevant to the political science student that wants to analyze this or the English major. Right. That wants to understand things. So I think it does. It's really kind of exciting. And I think, you know, my colleagues in the humanities and the social sciences are probably having lots of fun thinking about how to incorporate some of these tools there. I'm not having that fun because I'm on sabbatical, so.

So I know you have two minutes left until you've got a meeting to get to, roughly speaking. So just real quick food for thought. If there's any other things that Jason, you'd like to just put out into the thing, maybe we can do a follow up stream if there's any other ideas that we didn't get a chance to get to today. But thank you so much for being here as well.

My only comment is just to follow up on what Josh was saying. In chemistry, we have a long history of incorporating technology into the classroom. There's groups like the Molecule Consortium in the Mid Atlantic area that have been building computational chemistry activities using research level tools in general chemistry, and that's been ongoing for 30 years. And the same arguments for getting students involved with those calculations or you know, the same thing like an analytical chemistry course, getting them to touch the actual instruments that they'll be using in the, in the real world and when they leave the institution has a long history. So I think this is just yet the next iteration of using LLMs and using machine learning and using notebooks which have been around for quite a while, electronic notebooks. So it's very exciting to see how it all interconnects and just contributes to a long history of good chemical education.

With just a little bit of time left, we've got another audience question which I always love to see. So I'm going to throw it into the mix. The question was specifically about drug discovery, but I'm going to throw it out even wider. But saying for drug discovery or anything else on your mind, how do you think LLMs are going to change that?

Oh my. Well, we might need another live stream for that.

I'm definitely getting the impression from audience questions that we might need to do a follow up because there's a lot of interest. So.

All right. Just to give the TLDR version of this, there's actually a tremendous amount of work in using machine learning and even using LLMs to assist in drug discovery and things related to human health. You know, a few buddies of mine to mention, right. Include folks at future house, futurehouse.org, which is kind of an organization funded by the Schmidt Foundation. Just over the past weekend they used kind of an LLM rag based system to write Wikipedia entries on 18,000 human genes that didn't have a Wikipedia entry based on scientific literature. It does just as good as humans. They check. And now we have complete Wikipedia entries for every expressed protein in humans. That's an enabling technology for basic research. And the major pharmaceutical companies also have efforts in doing this. Some of our friends in New York City at Prescient. Right, which is a kind of a exploratory R and D organization that is part of Merck Come to mind. Right. As people that are really kind of using this type of technology to advance human health.

Wonderful. There are many a lot to look forward to. Well, thank you so much for your time. Thank you everyone who's joined us today. I'm looking forward to hearing more and reading more of that notebook. I know we didn't get to all of it. Thank you so much.

My pleasure. It's really, really fun being with all of you today.