Breaking Boundaries: AI CoScientist to Accelerate Science Research with Gabe Gomes, Professor at CMU

At 6 AM on the Sunday after that janky prototype, I woke up with this idea. Okay, there is a way for us to go from natural language all the way to the code that allows for running these experiments. And then Yousen sent screenshots and messages on Slack saying, AGI is here. And I'm like, you cannot joke about this. I came to the office right away to see what he was talking about. He had put together something that showed us There is lightning in a bottle here.

And I probably weighed out 4.5 milligrams of palladium acetate like 1,000 times over the course of a year. And I just thought to myself, man, this should be automated. You know, like, I am not learning that much. I would just dream of the ability to have a machine do that.

The idea of cloud labs is that you have, not so different from cloud compute, where you have let's say a warehouse, where instead of just running CPU clocks or GPU clocks, now you have hundreds of different types of instruments and hundreds of copies of those instruments, as well as technicians and robots that can perform the operations. I am so excited about what we're going to be able to do because we are not encumbered by, you know, having to worry so much about these things that took a lot of time but did not advance the knowledge. It's going to be awesome.

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. My guest today is Gabe Gomez, professor of chemistry and chemical engineering at Carnegie Mellon University and author of the recent Nature paper, Autonomous Chemical Research with Large Language Models, which describes the pioneering and highly influential co-scientist system that he and his graduate students built immediately upon the release of GPT-4. This episode was super fun for me for a couple of reasons. On a personal level, I studied chemistry in college, and as an undergraduate research assistant, my job was to optimize a palladium-catalyzed organic chemistry reaction for yield. In other words, I explored the space of possible configurations for the reaction to see which would produce the desired product at the highest rate. It was, to be honest, mostly brute-force grunt work, and over the course of that year, I spent most of my time Weighing out small amounts of fine powders. I used to joke that I felt more like a low-level drug dealer than a scientist, and only once over the course of a year did I observe an unexpected result that led to meaningful new knowledge. Often, as I worked, I would daydream about a future in which all of that could be automated. To hear how Gabe and his team have used GPT-4 in combination with a remote-controlled life sciences lab called Emerald Cloud Lab. To automate a significant part of this work, and to see how they applied it specifically to the optimization of a palladium-catalyzed organic reaction, was for me uncanny. If this technology had existed back then, I might have followed through on my plan to become a chemist. Far more importantly than my story, though, is how this paper, along with just a handful of others released over the course of 2023, represent the beginning of the AI-powered automation of science. While large language model breakthrough insights are still exceedingly rare, with appropriate prompting, scaffolding, and affordances, AI systems can now generate new knowledge. This is already working at a level that can create leverage for graduate students, and it sets the stage, in my view, for the next generation of models to do much more still. However long it may take to develop them, The advent of AI systems that can advance the scientific frontier is generally considered to be a major milestone and a critical threshold, both for the potential upside of accelerated progress in science and medicine, but also for the unpredictable dynamics that may develop as AIs begin to introspect and self-improve. Professor Gomez, as you'll hear, is extremely enthusiastic about the benefits and not too concerned about fast takeoff scenarios. But he is very concerned about the potential for misuse and abuse of these powerful systems. As always, if you're finding value in the show, we appreciate it when you share it with friends. I would suggest sending this episode to a friend in the sciences whose work might potentially be accelerated by a system like CoScientist. Now, I hope you enjoy this conversation about using large language models to advance the scientific frontier with Professor Gabe Gomez, professor of chemistry and chemical engineering at Carnegie Mellon University. Welcome to The Cognitive Revolution.

Hey Nathan, thanks for having me. It's pretty awesome to be here and listen to your podcast. And yeah, let's talk some science.

I'm excited about it. So you have recently had a paper, which I saw the first version of back in April, now officially published in Nature, which is obviously an exciting accomplishment for any academic group. What you are doing, I think, is really exploring one of the most important frontiers in AI today, which is the question of to what degree can current AI systems, and obviously we will think about this a lot for future AI systems as well, to what degree can they do science? This feels like a critical threshold to me, and you have shed some of the most valuable light on the topic. Congratulations on that work. And maybe you want to just start off by telling me how you did it so quickly from GPT-4 release in March to getting the first version of the paper out in April and having one of the papers that really has stood the test of time over the course of 2023.

Thanks for that. There's lots and lots to say here with regards to where machine learning has been, which role has it been playing in science and engineering and so on far before LLMs. With respect to this one in particular, what happens is my group had been working on fine-tuning large language models for scientific tasks. We started working on that in October 2022. And we were having a very hard time. We were trying to do things with GPT-2 and BERT and so on and fine-tuning those things for a few tasks that we haven't published yet. But we're going to be putting that out on symbolic understanding of scientific experiments and so on. And we were kind of frustrated with that. And then ChatGPT came out with GPT-3.5 and, you know, I started to see, okay, like there's quite a few things here that are quite interesting. I was at a conference that was very small and one of the amazing people that have been also working on this, Andrew White, now at Future House, he gave a talk. He didn't talk about any of his red teaming work. But he was very— it seemed like he had seen the face of God or something, how much he was interested in this. Fast forward to the day that the white paper for GPT-4 came out. It was March 14th, 2023, if I remember correctly. And that was a super busy day anyway. But during the day, one of my students and coworkers here and co-founder, Daniel Boyko, sent some screenshots of the white paper for GPT-4. And I honestly thought, it's like, is this a meme? Is this a joke? Or what is this about? He also sent the PDF. And I remember being home like 2 in the morning on my phone for some reason, just like this, sitting on the couch, reading the white paper and having my mind absolutely blown away. Specifically with the example of There was a task that one of the red teams did that GPT-4 had to try to convince someone to do something with TaskRabbit and was solving a CAPTCHA and said, oh yeah, I have a visual impairment. Can you help me? And it worked. And I was just like, this is insane. And then all the other stuff, like now this time knowing that was Andrew with red teaming in terms of chemistry and so on. all of the coding capabilities. And, you know, my group has been working, we are relatively, actually I should say we are a very new group. We started January 2022 at Carnegie Mellon. And it's a small group, but we have been working a lot on the idea of automation and machine learning applications to developing new chemical reactions and understanding and so on and optimization. I guess we were very lucky because we were already with a lot of these things brewing in our minds. And I started to tinker with it, with the API for fun. And I had like this absolutely janky prototype of something that is not even what ended up becoming CoScientist. It will be another paper coming out this month. And I was like, wow, this is really impressive. And we may be able to automate a lot of the annoying work that we do. And this, I guess it really comes at the core of it. I was obsessed with a problem that I thought we would face here, which is the following. CMU has the very first academic cloud lab in the world. And it's a facility that you can think very much as AWS, but for scientific experiments over 2 different types of instruments that you submit your experiment as code. And then it's performed by a mix of technicians and robots. And it really, you know, it's about $50 million initial investment that is here. This is done in partnership with a startup that started in South San Francisco from 2 CMU alum called Emerald Cloud Labs. And one of the things about this is that for you to write the experiments, you write this code, this fork of Mathematica and so on. And that was the thing that I was worried about, because a lot of physical scientists in biology and in chemistry, and sometimes also material science folks that are at the bench running synthesis and experiments, there is a barrier for them to go and code. And especially if you like now talk about something that is not as widespread as Python, for example. And I was worried that that would hamper adoption. So At 6 AM on the Sunday after that Friday janky prototype, I woke up with this idea. Okay, there is a way for us to go from natural language all the way to the code that allows for running these experiments. And I sent this message to my trainees, Daniel and Robert, co-authors of this paper. And I also sent this janky prototype that they ended up not even using because it was too bad. Like, I'm not a good coder. The next day, somewhere else on campus, and Daniel sends screenshots and messages on Slack saying, AGI is here. And I'm like, you cannot joke about this. And he's like, come to the office. And I came to the office right away to see what he was talking about. And, you know, in just this little, less than a day really, He had put together something that showed us there is lightning in a bottle here because we now can really leverage how to go from English to experiments. Once we saw that, it was nonstop push through to get this preprint, this first preprint ready. We had a lot of challenges. I was traveling at a wedding. And submitted the preprint on like the night before the wedding to make sure that it was going through. And it did. And it went. And my life has completely changed since then. The reason why we were able to be so fast, and especially because I need to emphasize this every time I talk about any of this work, the new Robert, they are absolutely fantastic, hardworking individuals. And the new really is— And so I'm sorry, genius. We didn't have a lot of the amazing tools, like even LangChain back then, right? It was just us putting together a lot of the tools that then would become straightforward things that people use nowadays, but they didn't exist. And we had to make our own things. And because of those constraints, we end up being able to make something that works very well. As you said, it stood the test of time.

The timescales on this are incredible. March 14th, as you said, original GPT announcement day. At that time, most folks didn't even have API access. Your preprint, I'm looking now, is originally dated April 11th. So you've got less than a month from the announcement to the first publication, which is pretty incredible. And I think just kind of a reflection of how fast everything in AI is moving, how much sort of a model advance opens up in terms of capabilities that are yet to be explored. It was another friend actually of mine who suggested, why don't you see if it can write Emerald Cloud Lab code? So we did try that a little bit back in the sort of September, October timeframe as part of the red team and reported like, hey, this thing does seem like it can generate pretty reasonable synthesis protocols for at least common stuff. My own chemistry knowledge stops at the undergrad level, and so I could only kind of push it so far. And the other limitation that we found, I think interestingly, this increasingly looks like it's to Emerald Cloud Labs' credit, I was not able to find any documentation online that I was able to use to inform that work. So I just asked it straight away in that early, early version to write Emerald Cloud Lab code, and it just totally hallucinated what that code would look like with a very realistic syntax. And this was one of my kind of early adjustment moments where it was like, okay, wait, but is this real or not real? I could tell enough that the synthesis of aspirin looked right. The stoichiometry or whatever seemed to a first read to be correct. But was this the actual Emerald Cloud Lab code? I really had no idea and I couldn't find any documentation. At one point in time you would've thought, well, who cares about that? And now it's like, actually, that might have been a prudent move not to put that stuff too far out there in the open. Hey, we'll continue our interview in a moment after a word from our sponsors.

At least when we decided to do this, there was quite a bit, if not all of it, of the documentation was online. But because we had been working with these folks already before, and for example, Robert in my team already had like done work with Emerald, he knew how to navigate this pretty well. But certainly it was online. And I remember because that was kind of a A little challenge that we had to parse things in a certain way so that we could do, well, it wasn't called RAG then, but it's called RAG now. Again, we were fortunate that in my lab we had these robots from Ultentrans. It's a small startup that makes very affordable liquid handlers from Brooklyn. And every single thing from Ultentrans is open. Open hardware, open software, open APIs. In Python, documentation is amazing. And we leveraged that to quickly do the initial test that we wanted. Turns out this is quite funny. So GPT-4, when it came out, was able to write some code for OpenDroneZ that would run. But because there had been some updates in between GPT-4 coming out and the version of open source APIs that we had, some stuff would like, would not run. And it's like, oh yeah, I just need to fix this, this, and that. It was like, wait, how can we update GPT-4 retrieval? And we, you know, we went on to do what now is RAG, but it was that, that showed us that's like, okay, we can really do this for Cloud Lab as well.

Fascinating. Okay. So let's take a minute and just make sure people have a sense of what it is we're talking about on a couple different dimensions. I want to make sure we sketch in a little bit more detail what Emerald Cloud Lab is and does and why that's valuable. Then we can kind of describe the system that you now call Co-Scientist that you set up and the different components of that. I want to get into what it can do. I've got questions about your kind of mental model of the different thresholds on the way to more and more AI-automated science. And then we can discuss kind of the likely implication of that for all of us. But starting with just Emerald Cloud Lab, I think this is a huge deal in its own right that almost nobody has heard of. And one of the things that you do, not to jump ahead, but one of the tasks that you managed to get co-scientists to do is to optimize a palladium-catalyzed reaction.

Yeah.

You could tell me more about this particular palladium-catalyzed reaction in a minute, but I'll first just tell you a story, which is that when I was an undergrad, I was this chemistry student and I worked in a lab for a year under Professor M. Christina White. And what we were doing was optimizing a palladium-catalyzed reaction. So I'm literally doing, as an undergrad research assistant, the same type of task that you have now got co-scientists to help with. What was ultimately the reason I got out of chemistry was how small the ratio of intellectual work to literally weighing out small amounts of fine powders was my time. I sometimes analogize, I'm not a big analogy person, but to give people an intuition for the sort of work that I was doing, I sometimes analogize it to the game Battleship, where you say, hey, you got a hit on my ship. And then the next thing you got to do is kind of explore around that hit until you can get the whole ship. Well, in reaction optimization, it's basically the same thing. When I got there, they had already got that first hit. They had discovered that there was a way to use palladium to catalyze this particular oxidation reaction. And Nat was like, okay, Nathan, you just finished your sophomore year. You don't know anything. Here's what we do. We do arrays of basically pick a variable, vary it, and run them all at the same time in the same warm bath and measure yield. And we gradually grope our way around through chemical space and optimize. But what was truly insane to me, and I sat there dreaming of something like Emerald Cloud Lab was I had these little tiny vials and these little tiny spatulas and these containers of powder. And I can still remember it, it was 4.5 milligrams of palladium acetate. That was the first reagent that would go into every little vial. And I probably weighed out 4.5 milligrams of palladium acetate like 1,000 times over the course of a year. And I just thought to myself, man, this should be automated. I am not learning that much. It was economical inasmuch as I was an undergrad. I would just dream of the ability to have a machine do that work. And at the time, those machines were maybe starting to happen, but they were not ready to displace me. Maybe just tell us a little bit more about Emerald Cloud Lab, because I think it is, even without the AI component, the ability to go in and program that kind of stuff to happen. The work that I did in a year, I mean, it seems like it's at least an order of magnitude compression that could have happened just with Emerald Cloud Lab, no AI required.

This story is really fascinating because it's a very common story for so many people that, you know, go into chemistry, go through that experience, and, and then, you know, think to themselves, why, like, why are things this way? Or maybe not, right? Because if you look at pictures of most of the things that we had in society, uh, let's say last 100 years, right? Cities changed, landscapes changed, how we work changed. And if you look at a picture of someone in a lab 100 years ago, and you look at a picture of someone in the lab now, it's not going to be so different, right? That is because so much of it comes to this manual labor that is just, as you said, sometimes not intellectually really stimulating. So Cloud Lab, the idea of Cloud Labs is that you have not so different from cloud compute where you have Let's say warehouse, where instead of just running CPU clocks or GPU clocks, now you have hundreds of different types of instruments and hundreds of copies of those instruments, as well as technicians and robots that can perform the operations that goes into those instruments. And Emerald is one of the pioneers. Emerald Cloud Labs is one of the pioneers in this, right? They're one of the first companies doing cloud labs. There are others. And I should say for disclosure, I am the AI scientific advisor for Emerald Cloud Lab. That is the idea that now you can have from anywhere in the world, you write the code for those experiments and they will be performed by either people for technicians for tasks that we do not have really good automation for it yet. For example, weighing solids. Unfortunately, weighing solids is a test that's a lot harder than it seems. Things are getting better, but we're not quite there yet. But like dispensing liquids, that's for sure, you know, something that we do not need people to be doing. We have many, many, many and affordable robots that can do that. This is what really leads to this compression that you mentioned in terms of how much time you would need to take to ask a question, right? The group that you mentioned, Christina White, is one of the best groups in the world on developing these types of transformations. They are critical to develop better therapies, better materials, a better and more comfortable life for us all that is strongly rooted in the fundamentals of organic chemistry. And yet, So much of it is so slow. And like you said, in the game of Battleship, so much of it is still, you know, got a hit. Let me try to do, you know, all that one factor at a time, or, you know, one variable at a time and try to optimize around there. This is something that I will say the more modern approaches to reaction optimization no longer go that way. They will use something like Bayesian optimization. This is something that many of us in the field have been doing for a few years now. This is also where machine learning really started to shine on experimental chemistry over the past, let's say, half decade to decade. You really touch on the nerve of the problem here and what has been bothering me so much. Imagine if you cannot go to the lab because you have some you have to take care of a kid or your parents, or you just cannot go to the lab. Does that mean that you cannot do science? Because I don't think so. Your intellectual focus here should be on thinking how to push those new reactions forward, those new transformations forward, not on weighing solids. And that is what Cloud lab and automation, I believe, strongly bring into democratizing science for many more people. And I, you know, I'm an organic chemist by training. And one of the subfields in organic chemistry that exists is total synthesis, which is where people essentially, you know, there's a natural product, they'll go and try to make that because that might have therapeutic properties too. Let's say, try to cure cancer, a type of cancer. The amount of work hours at the bench that these folks put in is astronomical. It's really amazing work, but it's extremely labor-intensive to the point that it slows down that progress and creates a culture that is just not really healthy or sustainable. So how can we change that? How can we really change that in a way that now we have machines that can do a lot of this grunt work? That's the idea of CloudLabs, in my opinion. And, you know, for the optimization aspects that you were mentioning earlier, machine learning plays a massive role now. Like it really does. I don't see how one can go and try to optimize a web reaction without using these techniques. That would be just— A waste of time, if I may put it that way.

I wonder how often— because I saw this too with Christina, Professor Christina White. The work wasn't on total synthesis, but the next group over was doing total synthesis. And so I did see some of the challenging work culture that you mentioned. And sometimes that really was dictated by the chemistry itself. You'd have these sorts of things where it's like, this thing needs to be cooled gradually over 12 hours. There's no law of nature that says that the timing of these reactions needs to line up with your childcare hours or whatever. So it often doesn't. And people were running reactions overnight, and you can imagine when it fails or it doesn't yield what they want, then they got to do it again. And so it does become a real— grind. I saw that up close. I wonder though, in the process of reducing this to something that can be fully explicit with code, I imagine there must be some percentage of the time where for some reason that doesn't work. There's some aspect of what is going on in the lab with the human, with the hands-on, where there's knowledge that is not even necessarily Fully conscious on the part of the human that is allowing them to make the reactions work. And then when you put it into code and you specify this is what's supposed to happen, hey, whatever, it's just not working. How big of a problem is that from what you have seen? Hey, we'll continue our interview in a moment after a word from our sponsors.

You're really hitting some really heart-touching questions here. And I'll say that this one in particular, Something that I have been thinking about quite a bit because I, you know, I talk about these things in a way that I want us to democratize access. And, you know, imagine if you never, if you didn't experience that lack of intellectual stimuli for waste solids, right? You probably would be a chemist still. You know, what is, we have been singing the praises of, you know, manipulating bits and For the past 20 or 30 years, like really worshiping at the altar of Silicon Valley. But man, chemists and chemical engineers, and especially nature, has been manipulating atoms, right? That's what catalysis is. And when it comes back to thinking about these tasks that perhaps are not quite automatable, Or perhaps there is a lot of intuitive knowledge that you don't quite know how to translate from, you know, the human way of doing something to now an automated fashion. That is fine, right? Like there will be time that will go gradually and being able to implement those things further and further. And this is why I say this because in the past, folks that were, you know, pushing automation and ML, chemistry tended to be a little bit too overconfident and push a certain hype that I really dislike. We need medicinal chemists that are extremely well-versed in what they do to continue what they do. But we also can simplify a lot of the things that are mundane and routine for them so that they can be faster at what they do. And with time, we'll develop better hardware and software that can accommodate for this. I do not expect in any way, shape, or form that we'll be able to totally automate certain techniques that, or even, you know, certain little aspects that don't go into supplementary information explaining how chemist XYZ did this. And in some other ways, you have folks like Professor Lee Cronin, University of Glasgow, and the hardware that he developed with computers and so on, that is more of an adaptation of how chemists have been doing this for so many years that is closer to how what you described as medicinal chemists do those tests would be. So there is a gradient here. Nothing needs to be black and white, and nothing is. And I do believe that this is important for us to think about The next steps for core scientists too, right? I am thinking a lot about human-machine interactions and a lot about embodied artificial intelligence, however you want to describe that, agents, in order to be able to perform more and more complex tasks.

So that's a perfect segue for me to describe what the CoScientist project consists of, and you can elaborate on this, but I think for folks who listen to The Cognitive Revolution, the general structure of an AI agent is probably increasingly familiar at this point. This was an early version of it, but it basically has become kind of the default structure. You look at the diagram in the paper and I've seen dozens that look very similar over the last however many months. So basically you have kind of one central— and I'm interested in how you think about it. Is it all one agent or is it like a team of individual agents? But you've got essentially one central planner module and then you've got its kind of surrounding 4 major affordances. And those are the ability to go on the web and get information, the ability to write and execute code, A la like an OpenAI code interpreter, the ability to search specifically through the documentation, if I understand correctly, for Emerald Cloud Lab. And then—

Other hardware as well. Not just Emerald, but like really any hardware that we give the documentation for can be implemented in a matter of either, it's from minutes to hours to a few days. Now, We are very good at it, so it can be done very quickly. Cool.

And then finally, the actual access to the APIs to make calls to the physical world. And that's one of the things that I think makes this so interesting too, is for people who sort of think robots or killer robots, or it's all very fantasy, here's a setup where there is already the kind of API-ified physical instantiation that's going to do the bidding of whoever's calling the APIs. And that API doesn't really know whether it's a human or an AI calling it. And probably, maybe increasingly, but certainly originally, it didn't have any need to figure that out because there were no AIs calling it in the early days. So I'm going to just make a guess and you can correct me if I'm wrong here, but as I understand it, basically what happens is The scientist comes to the co-scientist and you're kind of, I guess, taking inspiration from Copilot there, right? So this is something that a scientist can kind of work with in real time and get help from. And you say, hey, I want to synthesize X compound. And I don't know how much you have to prompt it or how specific the scaffolding already is in advance, but ultimately I want to send a program to Emerald Cloud Lab to synthesize this compound. Your job is to kind of take it from this natural language to those end API calls. Typically these agent frameworks kind of have a loop. So I'm assuming the first thing is like, well, let's think about that step by step a little bit. Let's maybe go do a search on the web for known methods of doing this, write some code to do the calculations for how much material we're going to require. We know that language models obviously are not great at doing stoichiometry multiplication. In fact, one of my reports as a red teamer to OpenAI about GPT-4 was that GPT-4 is the world's worst chemistry tutor. Because when I got confused and I was roleplaying with it, I roleplayed as a confused student and got the stoichiometry wrong. Stoichiometry is basically just The relative proportions of reagents or inputs that you need to have to get the right mix on the other side.

Relative proportion of inputs for your output here. I like that.

Anyway, when I would get confused, it would get confused. So running code is a really good way to clarify that. It understands the logic pretty well. I would even just do very basic stuff like CH4 plus 2 equals what and try to get that reaction to balance. It understands the principles, but it can get tripped up in the numbers, and especially if the numbers are gnarly, it's not great at doing that kind of arithmetic. So what it is good at is writing code to execute that stuff. So, okay, let's do some analysis, make sure we have the right amounts. Okay, now it's actually time to go. We know the steps, we know the amounts. Now we're going to— again, at each step, it gets returned to the planning agent. The planning agent sort of says, okay, cool, we got that update. Now we know how much. Now let's go look into the hardware documentation and see what, you know, what machine to use and what they're capable of. And then finally issue commands to the Emerald Cloud Lab so that it can actually do the thing. What am I missing or what would you want to elaborate on?

You got it. A couple of things that I would say is that we really thought about this to be in a very modular approach. Our initial intent was for autonomy. So we really do not want to have something that would be originally too conversational, but really just, you know, here's the task, go for it, right? And we went, you know, we went, we really went pretty wild with it, which was a lot of fun. You were correct about everything here. The things that I want to add is that in the execution of experiments, it doesn't have to be CloudLab. In fact, you know, it— Often in our lab here, it is not. It's just whatever hardware you have available to run the experiments that you want to run. And one of your questions was, if I think about it as a collection of agents or one agent. And this is also something that I see people getting constantly confused about. The capabilities of large language models because essentially they think that, you know, if you go to ChatGPT or Bard or Claude and you ask for this task and the, you know, chatbot or the LLM that you use cannot perform the whole thing end to end, that is not good. And that's absolutely not the case at all. When you start to think about even how we think in terms of system System 1, System 2, and so on. Like, we have to have some way also of trying to help the LLMs, like, process information that way. So when we started this, one of the immediate things, even with my janky prototype, was that you would have something that would be a planner and would have something else that would do other tasks. And of course, I'm so happy that you brought up the problem of our semantics and so on, because That was one of the key ideas that Danilo brought up, which was giving it the ability to run code in a Docker environment and so on would allow for us to do all simple math without having to worry about the shortcomings of LLMs. So you have the planner that is an LLM, hopefully Well, not hopefully, but ideally that's going to be the strongest LLM that you have access to. And then the documentation search and the web search, also LLMs. Many of our initial prototypes really tried to use GPT-3.5, for example, for those tasks. And we had some shortcomings. Maybe nowadays with Mixtral or other really powerful open source Llama 2. Maybe that would not be an issue anymore. And the other 2 components, code execution and experiments execution, are not LLMs. They're tools, right? In this process, the whole thing is co-signed. Now, I would be remiss if I didn't acknowledge the work of many other amazing people working on this, specifically Andrew White and Philip Schwaller. Philip Schwaller is a professor at EPFL in the Department of Chemistry. He's a great friend and collaborator now. Literally today, one student from Philip's group arrived at my group and is going to be with us for 6 months. And Andrew is also a friend, mentor, collaborator. They had ChemCrew that they saw it as showing the capability of LLMs to be able to use tools for chemistry applications, chemoinformatics, and so on. We released the 2 preprints back to back on arXiv together. And you can see that, you know, our approach that then we didn't call co-scientist, call agent or something, was we want something that will be autonomous. And now we have, you know, chat mode and so on. It's many other things. This is, you know, how I see it. There are many things that it didn't make to either the preprint or the final paper that is absolutely fascinating to see, especially if you are an experimentalist, a scientist that, you know, for example, when we prompt scientists to perform those complex palladium-catalyzed reactions, the same thing that you and I that never have, like, may not have never done something is that you will start to search the web first with, I don't know, Wikipedia, like read it. It's like, what is this thing? And it Keeps digging, keeps digging. And the planner in, well, let's say conversation with the web searcher will have that back and forth of like, I need more information and, or no, I'm good now. And once the planner is satisfied, quote unquote here, then it goes, okay, I need to perform the experiments. This is what I have available in terms of hardware. Let's say it's the OpenTrons OT2 in Gabe's lab. So I need to go and now do the calculations for volumes and whatnot that uses the code interpreter, code execution, I should say. There wasn't code interpreter back then. And we'll do the calculations. And this is the part that I find so funny because sometimes you will make little mistakes with variables. Python will give an error. It's like, okay, corrects that, makes another little mistake, corrects that. Once it gets all good, then Pass that on to the planner back, and the planner now knows, quote unquote knows, that it should use the documentation searcher or communicate to the documentation searcher about what functions and what it needs to write the code to execute those experiments with the code execution. That is the workflow for certain tasks that you can have analysis of results. On the fly with this, that's where things get really beautiful because now the planner takes those results and can say, oh, you know, this didn't work as expected. Let's change this, this, and that and go try again, right? Not differently from how you did as an undergrad. And a lot of scientists do research in their, you know, how I did things up until not so long ago. So I really do see this massive change now where you don't need to learn the exact nuance of the code that goes into Emerald Cloud Lab or know everything about the Python APIs for Opetrons or Hamilton and so on. No, you need to have your idea of what you want to do and that's it. That is what is so powerful here, in my opinion.

Let's talk about the power a little bit and then I want to get into a little bit more of the details. The power meaning, what can it do? There are 6 things, 6 distinct results listed in the paper of things that this co-scientist has been able to do from, if I understand correctly, just kind of a relatively brief natural language prompt like synthesise aspirin or whatever, to on the other end, aspirin comes to exist in physical form in the world. I'll read the 6 and you can kind of comment as we go wherever you want. So number 1, planning chemical synthesis of known compounds using publicly available data. That would be like your aspirin synthesis is well known, go find it, Calculate the ratios and go. I called that text-to-protocol.

Nice. Yes.

Okay. Number 2, efficiently searching and navigating through extensive hardware documentation. We've kind of covered this, I'd say, already, but just to emphasize, for folks that have not spent time in chemistry labs, there are a lot of different machines and they have all sorts of different buttons on them and all sorts of different details, right? Even for machines that may have Conceptual similarity. There's a lot of different, you know, where does where does the sample go and what are the requirements? What sizes? How much? Blah blah blah blah blah. I had a mercifully limited crash course in that stuff myself, but a lot of machines. Anything to add on that?

One of the things that when building coscientists as the as what we wanted it to be, we never intended it to be to be something for chemistry only, right? That is a reason why. I decided to call it co-scientist, not co-chemist. It's because indeed we are able to do a lot more than just chemistry. It's totally field agnostic, which means that it goes back to these 6 tasks, let's put it this way, that we demonstrated in the final version of the paper are simply to say that are just examples. And we decided to focus on chemistry because we are— chemists. So, you know, to show these experiments, but I can tell you that we have done a lot of things that are not published. They are in biology or material sciences with scientists that, you know, we had absolutely no problem with it. Now, going back to the hardware documentation, it's not just hardware, but also software that is very, you know, specialized software. that people will take years to master. And now, no, you just English and it works great.

Okay, cool. So yeah, let me read the list again, but also this time I'll kind of abstract away from the chemistry context a little bit. So planning chemical syntheses of known compounds, that's in general kind of orienting yourself to the common knowledge of the literature, the current state of the literature. The next ones are all about these kind of tools, whether it is searching through documentation, hardware or software, issuing high-level commands. Notably, you report in the paper that one of the machines that GPT-4 was able to use was released after the training data cutoff, so you could be confident that it was figuring out how to use that machine, Based on documentation that was just provided in the context window and not relying on its pre-training, which is an interesting distinction at a minimum. Definitely has some implications, I think, for AI proliferation as well. And to what degree do you have an ability to predict what that's going to mean in the future? Low-level commands to the machines as well. So kind of high-level, more conceptual type stuff, but then also down to the microcontrolling 5 is kind of putting that all together into complex tasks that basically combine multiple aspects of the first 4. And then the 6th is optimizing. And this is the battleship type work that we've already talked about. That one seems to me like where you actually get to the level where you're displacing me as the undergrad research assistant.

Hold on, hold on. Let's talk about the first 5 and separate from number 6 because number 6 is quite different from the rest, I would say. So the first 5 are essentially, in my view, is how can you have this very natural language, for the lack of a better term, approach to how we think about doing these tasks that as we mentioned earlier, are either very labor-intensive or require a lot of qualification or a lot of time to be learned, right? And that is, you know, that is all, I would say, in the realm of everything that are maybe not even state-of-the-art anymore in terms of large language models. And that on its own, to me, is a massive change because I can't now, in terms of cognitive revolution, take a lot of the cognitive load that would be used to, you know, oh man, how do I like do this call to this API in this way and so on? And think, how can I actually think in terms of breaking and forming these bonds? Or how can I actually think in terms of doing this DNA split for whatever task I'm trying? It's true that We were very adamant about making sure that it could generalize to things that were not in the training material that goes into GPT-4. And RAG really plays a massive role in this. And it's fun to now have a word for it, which is Retrieval Augmented Generation, but back then, we didn't. Now, Point 5, which is the one that combines different aspects and different hardware, to develop a workflow, a plan and a workflow that then you can tell CoScientist to do something and it will autonomously plan, design, execute, and analyze what that experiment, those experiments and those results come in. That is where, you know, that things really gel in terms of it's much more than just I give English, here's code, right? Because this is a part that, let's say you have a good idea. Let's say you are a chemical biologist and you are working in trying to understand how certain cancer cells grow. And the way for you to do that is by using a fluorescent tag on that. But the chemistry to do that conjugation, to put the fluorescent tag on the surface of the cancer cell is something that you do not know how to do yourself. No bother, because you've read the literature and you know that there's some ideas out there and you can ask Co-Scientists, hey, this is what I'm trying to do. Let's go. And Co-Scientists can either be in a conversational mode that will assist you even in the lab without robots how to do that, or in an autonomous mode if you have the hardware for that. So now a biologist that otherwise would have to wait until someone to develop a whole new protocol or, you know, kits to do that conjugation can do that instantly, or as instantly as the experiments will allow, to continue their work on tracking those cancer cells and advance their research. The time that you already saved is tremendous. And you can think about the flip side, which is, you know, imagine a chemist that knows how to do all of the synthesis but may not know how to work with cell lines and so on. Training is complicated. Surely they can go and learn this or rely on cloud labs and so on, but they also can use co-scientists to assist them in the lab and try to do those experiments. This is what I see also as an important democratization of science that we are not paying attention to in how AI can— modern AI can assist us. And then we have point 6, which is solving these optimization tasks. That is something that my group is very focused on, developing and optimizing complex chemical reactions with catalysis for developing materials that, or molecules that are part of materials or therapies and drugs and so on. And this is where a lot of chemists in let's say in pharma, for example, will spend a lot of time developing to both make— to scale those reactions and to make them as cheap as possible. And don't forget that there's a lot of variables, as you mentioned earlier, that goes here. So for many years, people tried to do these kind of experiments and one factor of time and so on. And this grew and grew and grew. And Gaussian processes, of course, Bayesian optimization is one of the most common approaches to this. My group had been working on implementing more and more chemical information about the mechanism of the reactions into the kernel that goes into Bayesian optimizers and so on. But we thought, well, what if we gave co-scientists this task? But in order for us to actually do this well, let's write it as a game. So we really took inspiration from all of the reinforcement learning approaches that folks at DeepMind and others had done about this. The game is the following. You have 20 iterations. Your goal is to maximize something, in this case yield, and to simplify, the metrics, we use normalized advantage, which is just a different way of doing the math for yield. So with that goal, here are your constraints. You have access to this list of chemicals and this list of variables. So for one of the experiments that we had, or computational experiments I should say, that we had in the paper, it had over 6,000, about 5,000 5,500 possibilities. And if you were to ask a good chemist to do this, they would probably first see what results there are available, what kinds of things are similar, and then start to poke around and see what is possible. More modern approaches like we've been developing and what, for example, the NSF Center for Computer-Assisted Synthesis I'm part of, does is to use those modern techniques, data analysis, Gaussian Bayesian optimization, and so on. We thought, well, what if co-scientists can just do this? The task is to optimize the reaction. And here's the list of what you can do. Here's the game. And we let it go. And we did this many, many, many, many times over. This one figure in the paper, the last figure, It was by far the most expensive experiment, computational experiment that we've done. It was like over, I don't even remember the numbers, but it was by far, far, far the most expensive thing we've done because we really wanted to show that this was, we wanted to see if it was possible first and foremost. And not only it is, but it also has some advantages that, to be quite honest, I didn't think about until we were doing it, which is when you do Bayesian optimization and other techniques like that, you do not have a very good a priori reasoning for why the model is picking this point in the fitness landscape. And that is fine. You can retroactively go and understand that if you have enough domain knowledge of the subject. But what Quant Scientist offers you is as simple as, let's do this because of this, this, and that. That reasoning that we call, and I'll put in quotation marks, chemical reasoning, is something that's really, really useful because now it says, well, let's choose an additive that has a group that is more electrowithdrawing than the one before and keep everything else the same. What, very similar to what you did as an undergrad, right? And it will go and perform that and see, okay, the yield went up. So in that moment, the planner now knows that this is something that helps in improving the yield. It may hit a bump, a threshold or anything that very much like you and I were to do this would But it's extremely good at the strategies that it puts forth. So of course you can think about combining this with some of the techniques we talked about and so on, but on its own, it was quite impressive. And this is again, another thing that I brought up that you also mentioned. We don't need to be spending so much time at certain tasks that are often not cognitively interesting or intellectually stimulating because we can essentially now put those aside for co-scientists or whatever to do them while we're thinking about other more interesting, impactful things. That is to me what will help. So I do not see this as, you know, removing anyone from workforce or anything along those lines. I think as this is a tool and it's a powerful one that can really be helpful in accelerating how we do science.

Let's talk for a second about all the different variables that the optimization process can include. Then I want to talk about a little bit of just some of the techniques that you used to build this thing. And then we can come back to the results and implications. In terms of just optimization, it's a very high-dimensional space. I'm just thinking back to the one reaction that I was optimizing, you have which chemicals are you going to use, how much, at what point are you going to add them in time, what's the temperature going to be, what's the pressure going to be, what's the light conditions going to be, are you stirring it, when are you stirring it, how much are you stirring it. Going back to what chemicals, that's a whole rabbit hole unto itself.

Right.

In my thing, I made one very minor contribution to science over the course of that whole year. I guess if you want to give me the Edison credit of finding all the things that didn't work, then I could claim more discoveries of things that didn't work. But in terms of new knowledge of things that did have useful— that actually moved yield forward and it was like a conceptual discovery, it was one in the whole year. So just incredibly vast space. And I have to say, I would guess that GPT-4 is better than I was at kind of guessing what those next trials should be. I had the help of my advisor at the time who was like— and specifically the one thing that I found was we had a couple of different versions of the reaction. In the one version, it was had already been discovered that more acid would push it forward toward higher yield. But then in the other version, we kind of were working under that same assumption and tried to do an experiment to kind of demonstrate that. And the slope of the line was actually the other way. And so I printed out the result and I was like, huh, it looks like it's going the other way. Have we tried less acid? And sure enough, So my advisor was like, well, here's what we should do. Let's try the least acid that we can try and see how that goes. So we tried it and sure enough, that was my one contribution to science. In the other version of the reaction, less acid was better. So anyway, it's just a huge space with all these different variables. 20 iterations is not a lot. I probably did, I don't know, 5 times that much over the course of a year. And again, I was just working on this part-time. So a grad student would probably do hundreds of these over the course of It does seem like GPT-4 probably is better than I was. Is that wrong?

This is great. First of all, what you said about the Edisonian approach, that you did a bunch of stuff that did not work, that is a problem with science because we need negative data.

That stuff didn't make the paper either, by the way, of course.

Exactly. We need to change the culture of physical science and engineering to actually always include like negative data, because, you know, in doing, for example, Bayesian optimization campaigns, negative data is just as important as positive data, right? Like it tells you, it's like, hey, that part of the landscape, no good, like go somewhere else. 20 iterations indeed is not that many. And we did that on purpose. We really wanted to constrain this, right? And we did a comparison with some not True, true powerful Bayesian optimization, you know, Cositus was either comparable or better to the BO that we— the Bayesian optimization that we used. So I have a bit of a hard time saying, oh, you know, it's better than GPT-4, this one's better than person XYZ at this, because, you know, we can see that, but then—

Yeah.

Sometimes it just makes mistakes that are just so stupid for different things. So these are very powerful, intelligent machines that sometimes make absolutely stupid mistakes. And, you know, I really want to emphasize that what we built here is not a silver bullet. It does make mistakes, obviously, and it's far from perfect, and there's a lot of work to be done, right? But it is pretty good at picking variables and how to modify them. Going to what you were listening earlier, right? That you said, you have which chemicals and how much of this, what temperature, how long, and so on. So you have categorical variables, right? Like which of this and so on. You have continuous variables. So like, what's the temperature? You can vary that continuously. And you have ways of taking what are categorical variables and transform them to a space that makes it look continuous. And then you can do these, all of these nice optimization strategies that you can take nice gradients through and so on, right? That's the kind of Bayesian optimization that we do that then also incorporates information about reaction mechanism. That is the differences here. And we do not have coscientists do any of this. And if you were to think about something like, I don't know, let's say we had $2 million for no reason that we went to OpenAI and said, let's fine-tune GPT-4 for these types of experiments. And it probably would become something really, really powerful that you wouldn't even need to do a lot of the other things that we do to make sure that hallucinations are not an issue, that you can retrieve and so on. I like the comparison, the point of ChemCrow with tools. And we also give a lot of different tools, something that, as a disclaimer, Daniel Boyko and I are co-founders of a company that will That will be building a version of this that will be available, commercially available with all the guardrails and all the bells and whistles and a very nice interface. And in thinking about this, we start to give it more and more tools that did not make into the paper because of, you know, how much we've been working on. And suffice it to say that we know now how to give it different tools for different Problems that on its own GPT-4 is not super good at, but once you give it access to these tools, then it's no longer an issue. In the paper, we had examples of how it wasn't super great at certain very simple molecule synthesis that any organic chemistry undergrad would know how to plan. That's okay. It really isn't a problem because there are so many amazing tools out there for retrosynthesis, some of them developed by Conor Colley at MIT or Philipp Schwaller again at EPFL, that rely on transformers as well, that can be used by co-scientists to do this. So it's not like a world on its own. And that is the thing that brings back to even your question earlier, is this one agent or many agents? We really need to take a step back and think about not just one technology, say LLMs, or another, say automation, but how those 2 things coming together are much larger than the sum of the parts. That is something that I sorely miss in a lot of the conversations that we see online, unfortunately. And what I seek is really like seeing how those things coming together build the next, that give the next step on that. And that, you know, brings to points that you, that are more technical, right? Like how is prompting? How is scaffolding? How is the tool use and memory? And so on. These are all things that you, if you were to take a trip back to March when GPT-4 came out and said, hey, by the way, there will be a way that you can retrieve memory. People would be mind-blown. And now it's like, yeah, it's fine. No big deal.

I think the number one profile of a person that listens to this show is the AI engineer. These are folks who are working with the different paradigms that you were an early developer of, with agent scaffolding, the loop, the RAG, the tool use, the memory. As you said, most of that stuff was in either nonexistent or very immature form at that time. Now it's come a long way, but I wonder if you have any kind of tips or big lessons learned. Were you even using a vector database in the early going, or were you just kind of using keyword search against existing search? Anything that you think stands out as a big lesson or things that have gotten easier, things that have just kind of changed over the last 8 or 9 months since your pioneering agent setup to today's agent standards?

Isn't it amazing that not even a year ago we did not have what now is a profession, the AI engineer, and now we have it and it's making such a massive difference in the workforce and how even we think about things, right? Like how, what is possible. I used to think about, you know, oh, we want to push the boundaries, but we always need to know what is feasible, what is possible. I no longer think that. I think that we can do, if we can do something, if it's doable, then it's also feasible. And that, you know, is something that these superpowers that the LLMs gave us. And going back to when we started to work on this, because all of the really nice tooling were not either not available at all or not widespread. And because we really built all of the initial frameworks at that point, we do not rely on many, too many things. It's incredibly vanilla, um, in some ways with regards to all of the amazing tools that folks have developed, like, uh, LangChain. And I remember when BabyAGI came out and I was like, oh, this is interesting. Like, there's a lot happening here. And of course, you know, at that point folks were very disappointed. It's like, oh, you cannot do everything on this. I was like, yeah, but like, look at the ideas, right? And more recently, if you look at what Microsoft put out with AutoGen, for example, that's a language agent. It really is pretty awesome. I really want to emphasize that knowing your tools is obviously very important, but it is not the end-all be-all of anything. I really prefer to think more in terms of what are the goals? That you're trying to accomplish. We are incredibly agnostic to what the tools we're going to be using. This is something that I, we're really proud of ourselves in and it doesn't matter, right? Like, so we didn't have, there was no vector database that were as good as the things that we wanted to use that you can use now. Okay. Let's figure out how we can do this with something as simple as like just NumPy. And, you know, you can do this. Like, it's not— there's a lot of these things too that look much more complicated than they need to be. And that is something that I really want to push on. Like, you do not need to do the most sophisticated piece of software in order to get stuff done, right? And I say this honestly as being guilty of, as someone that always wants to push for the most cutting edge. And my trainees paradoxically are the ones that are much more pragmatic than you really is like, no, you don't need that. Just use this simple thing that will do the work and it will work. It's like, okay, yeah, you have a good point. So for the version that's out and so on, of course, scientists, if you want to take a look, We know how to do tool usage and many others then showed how this can be very powerful and like mainly benchmarks, et cetera, et cetera. Similarly for RAGs, similarly for memory. RAG indeed was quite the challenge. There was not the name, right? So, but we had a very problematic situation. In dealing with one of the documentations that we're dealing with because it was very comprehensive and very wide. And this is where the weirder things come together. So Danilo, when he was doing his master's in organic chemistry at Moscow State University, he was also an ML engineer at VK. And he worked with a lot of different things, but he really did also do work on search and computer vision and so on. And that knowledge really came in handy because he knew how we could do searches across the, you know, what is not massive space in comparison to really anything that you'd think in terms of search engines, but large enough that was a problem for the design of our workflow, our platform. So that was one thing that he was able to pull together from things that he knew before. We exemplified that with 2 different algorithms, but in the end, we have one way of doing things that really works best. Prompting is something that is also very, very, very Very powerful. And there was the rise of prompt engineers. Of course, we saw that. And what we see here is that prompting comes very, very closely related with both tool usage and like how good or bad that would be and with retrieval augmented generation, right? So we did put a lot of effort into that as well as doing lots of A/B To that testing in order to make sure that we would get consistent results to what we needed. Then we also do a little bit of a mixture of experts in some way to make sure that things are smooth. Let's put it this way. My tip would be think about the goals first and don't get too hung up on the tools because if you— actually, we're a bit cynical saying this because the reality is that A lot of the ideas that came to mind came when GPT-4 came out, and that's a tool. But once you have that, you know, just go for whatever goal you're trying to achieve. That's how we try to do things here and how we've been doing things. And you do not need the most complicated set of tools to do very interesting work.

I mean, that resonates with me in a totally different context. I won't I won't tell this in long form again because people have heard it, but with the Waymark project of trying to write good video scripts, I find, and ultimately create good videos, which is a multimodal challenge, it is almost like you cannot remind yourself often enough to pull back, come up for air, and be like, what exactly was I trying to do here again? It is so easy to get lost in the details of The tools and the evaluations of different techniques. And obviously all the evaluations end up being imperfect. And there is something really important, and this part is still a little bit beyond the AIs in many cases, which is notable, to just do that sanity check and make sure you're not kind of autoregressing down the path. It's easy to do that if you don't have that kind of system interrupt and just reminder of what the goal is. That is more of a cognitive tip perhaps than a technical tip, but I do think it is a really important one for AI engineers across any number of different domains. One thing that definitely stands out is that context window has exploded dramatically since the early work. Has this changed or would this change how you would think about designing a system even at the level of one agent versus 5 subagents making up an agent? I sort of imagine a chunk of that division is a function of context limitations and the need to manage context. Maybe that's wrong, but it's at least part of it, right?

Context windows are something that there's some good papers showing that depending on the models, I remember showing that there's too much attention being paid to what's at the beginning and the end. What's in the middle gets kind of like, yeah, whatever. And what I can say is the following. It would not, as of like today, it would not change how we design in terms of this multi-LM blocks. And so what has changed is for something that we haven't published yet, but it will be coming very shortly, that imagine that you want to now leverage the power of quantum mechanics with computational chemistry for designing new materials. And a lot of those softwares are incredibly tedious to learn and so on. That is where I started. And that is my original field too, computational physics, organic chemistry. And we're going to be showing soon how this all can be simplified significantly. And that's where context windows played a much more important role than with what we did here for CoScientist. I do want to emphasize the fact that we did this distribution of work a lot, thinking in terms of how, well, you know, Kahneman has told us in terms of system thinking, fast and slow and so on. Take that into consideration to break things down, to simplify how you tackle problems and also try to help your agents to tackle problems, like trying to cram everything into one instance. Why? Many cases you do need that, but in many cases you do not. And that turns out that when you can break things into planning and execution and so on, it turns out to work a lot better than if you were to try to cram all your needs into one massive context window. The context window stuff is fun for very large corpuses, right? I don't know, you have vector databases that can also assist you with that and it works pretty great.

Well then let's get to finally the implications of all of this. One question I have is how you conceptualize it. I'll offer my mental model and then you can give me yours. But I think of AI very often in terms of Threshold effects, meaning when an AI gets above a certain threshold, the situation is qualitatively different than when it's below that same threshold. I sometimes call that the can-can't boundary. And that's also kind of bound up with notions of emergence, which have been certainly in the air again recently as the emergence paper won the NeurIPS Paper of the Year. There's one threshold here that we've clearly crossed. From the results of the paper, it seems pretty clear that GPT-4 can do this optimization, whereas 3.5 cannot. And I've seen that in a couple different cases. I talked on a Research Roundup episode about the software improver that then acts on itself and becomes the self-improving software improver. And that had a similar thing where GPT-4 could improve, but 3.5 actually just got worse iteration after iteration. They both kind of leveled off, but GPT-4 was leveling off above and 3.5 was leveling off below. So it seems like there is this kind of divide where it's like if you're on the one side of it, things can kind of roll downhill in an improving way, and on the other side of it, it's not happening. What mental models do you bring to this whole scenario of AI in science. Another one that I have in mind, by the way, I think you kind of alluded to, is when can a general AI train a narrow AI to solve problems that it itself cannot solve? So we know, for example, that AlphaFold can do things with protein prediction that GPT-4 cannot do and that no human can do. But at what point can a GPT-N design and train an AlphaFold to solve whatever problems it needs to solve. I don't think we're there yet, but that certainly seems like another big threshold. Anyway, you can explore that any way you want. You can challenge my threshold model. You can list out some thresholds you think are important. But I'd love to hear how you think about this kind of escalation ladder that we seem to be on.

If a tool already exists, then GPT-4 can go in. It will be able to use it like AlphaFold. This is something that we are doing now very, very, very routinely. Let me just stop there. Okay. We have a project that we call Netflix for Enzymes. So that's what I'm going to start with. When it comes to the differences between 3.5 and 4 with what we immediately saw here, there might be something about smaller models that Would be better at following instructions. Like one of the things that we saw that was quite a bit of a problem was JSON for 3.5 that, you know, for 4 was not a big, not a problem at all. Yeah. This is the thing with extreme scaling and emerging capabilities, right? Like a lot of this we don't know until you have the model and it makes the whole field to be fascinating because, you know, like what is going to come out of this?

Sure.

There's a lot of talk about, you know, are we at the limits of the architecture? Maybe that's not the point. The point is for physical sciences, we are absolutely nowhere near having anything that has extreme scaling the same way that large language models. And I say physical sciences, I'm biased. I'm really talking about chemistry. Think about the cost of one image, for example, for vision tests.

Yeah.

And computer vision models and generative imaging and videos and so on. What is the cost of one image out of the whole internet versus the cost of one experiment that you do for chemistry? That gap is enormous. We are trying to solve that gap nowadays by trying to pull together what we think it would be an image that moment For chemistry. If we're going to get there, I don't know, but we're trying to. Because I do see that not only with very general models versus narrow ones, at some point we're going to be able to have GPT-whatever to, like you said, just build an alpha fold from scratch if you have enough. Compute and resources and use that to perform tasks that would be useful for. We do have that in very, very small early days for very, very small models that are like regression or classification models for things that we care about in what we do in my group. But I would never say that, you know, similar to What you were mentioning earlier, that's like this software that improves itself and continues to self-improve. Nothing like that at all. This is awesome, right? Like it really is. I see no limits for what we can do. Now the question really is, what are we going to do with this? Because all of a sudden we have this alien technology, it's some ways that gives us superpowers. Why are we using this to, you know, make things that might be entertaining but are not life-changing? Why are we not using Claude to help develop the next therapy for whatever, right? I'm sure there are people working on this, but I want to see much more of that and much less of yet another AI girlfriend or whatever like that, which is important too. People need companionship. But I'm trying to say is that there is so much that can be done and needs to be done that AI engineers, please, like come and like help us, physical scientists and engineers that do not know enough, that need your help to make sure that we can do the best applications of physical sciences, biology, chemistry, and material science, and chemical engineering, electrical engineering, so on, with AI applications that are fantastic. So here's what I talked a lot about democratizing knowledge that otherwise would be difficult if you're not a specialist. And something that we had in the preprint very clearly, boldly, given how things have moved on, I am very optimistic about, is on also the potential for misuse and dual use. It may not be something that people like to hear, but we must be responsible and transparent and honest about the fact that these technologies are enabling, and it's a tool. And, you know, it's not the fault of the tool That can be used for unsavory purposes. But we must be vigilant and we must be responsible to acknowledge that and make sure that we mitigate the possibilities for harmful applications, not only in the context of what my work is, but also in societal applications that we are not discussing in the context of this episode. And when we had the EO, the executive order from President Biden in October, that's, you know, what I saw as being a very good path forward, as well as the UK security summit and the one that will happen in France and South Korea. And all of the movement around this is really nice. I'm not saying in any way, shape, or form, I really want to emphasize that The only way for us to go forward is for us to develop these technologies fast, really. The technologies that we're developing here, that OpenAI and the other big AI labs will help us mitigate the possible dangers. I have no question about that. All I'm saying is let's do it responsibly.

Yeah, this is definitely, I'm very much of two minds on this, where I'm a major enthusiast And love the upside. And then I also am like, geez, this does seem like we're getting potentially close to some thresholds beyond which it seems to become very difficult to predict what might happen. One, it's affectionately referred to as PASTA, the Process for Automating Science and Technology Advancement, is one that the AI safety community has held out for a long time, in AI years anyway, as sort of a threshold beyond which not only do things change just generally potentially very quickly, which is potentially good but also can be challenging, but also that if systems get past the point where they are now advancing state-of-the-art knowledge autonomously, then that can also be kind of folded back on itself and creates the possibility for these sort of positive feedback loop runaway type scenarios. GPT-4 is not going to run away, but a system that can improve chip design and improve architecture and kind of establish its own TikTok type of dynamic, at some point it certainly seems plausible that it could.

I'm not worried about that, at least in the now at all. You know, a lot of those count with what-ifs that we, we as a community simply do not know. What I, I worry much more about things that are already out in the world that can be used. And I also want to really strongly emphasize that there is, it's my strong opinion that we We should not try to curtail any developments or really stifle ideas in large language models and any of the other path technologies around these because it's all about intent. Nothing on its own is bad or good. Tools are not, but it's about intent, right? There is a lot to be compared here. Not to the usual thing, in my opinion, that folks will say, this is not like Oppenheimer and folks back in those days. I see this more similar to the idea of, for example, gain-of-function research that is very problematic in certain circles, but it's something that can assist with understanding what is possible in terms of Dangers. Not saying this is what we should do. I'm just saying that there are frameworks that can be done. And by the way, I'm not a policy person, right? Like there are policy people that are very smart, that work really hard. They are dedicated to this. There are AI safety people. There are AI ethics people. There are ethics philosophers. Those are the specialists. All I can hope to do is to provide information for those folks in the cleanest and most approachable way so that they can be informed about what needs to be done.

I conceive of my role pretty similarly in that the scout notion is predicated on the existence of generals somewhere who hopefully I can help inform. So help me understand what your outlook is. I think the whole spectrum of AI risks is worth thinking seriously about. In the original paper, as you said, there was a very prominent highlighted statement of, we're not releasing this prompt, code, et cetera, because there's no guardrails and we are not really comfortable with that essentially. And that is in a smaller font in the final paper, but it's basically still there. As I understand it, the details of this underlying implementation has not been released. Do you have a scenario in mind or conditions in mind at which point you would say, okay, now it is cool to release it? I'd love to hear more about how you think about— because there's obvious tension between democratizing on the one hand and releasing and under what conditions. Do you have a scenario in mind at which point you would say, okay, cool, now I'm satisfied we can release this and it'll be— It'll be fine.

In the development of the original that went on to be the free print, we saw these possibilities and kind of, you know, like you could see a hint from the red team work done by Andrew White. And we pushed that to the very limit. And we saw that we ought to be responsible but not alarmist about The possibilities here, and a lot has changed since then with respect to how company, the big AI labs are looking at this, how governments are looking at this. But I will say that for quite a few months, I had really hard time sleeping because I was worried that, and I remain worried about about this to be. Quite frank here. And I think that a lot needed to move a lot faster than it has, and it hasn't. And a lot of it has, like, especially when it comes to government policy, moved a lot faster than I could have thought, really. And I wish there was a stronger discussion with some of these companies. Well, we really love these models and the capabilities, and it's precisely because the upsides are so strong for everyone that we need to be responsible about the potential downsides so that, you know, Like, imagine if one unsavory case happened and we were to just push away all of the good sides. I don't like that at all, right? So how do we mitigate that? I don't know. And it may seem like I'm avoiding your question, but I'm not. The point is that I do not know how to answer your question in a way that I would be comfortable with that as of now. We have a version of it on GitHub and it's open. Similarly, ChemCrowd also has a version of what they did on their GitHub and it's open. But a lot of the things that would make me most worried are certainly not available anywhere. This is something I take very seriously, that we take very seriously. I am happy to see a lot of the discussions that happen around this. I wish folks also were not so dismissive about certain possibilities simply because they make comparisons such as, I've seen XYZ in cybersecurity in the past and this is exactly the same. It is not. Or also people saying that this is absolutely the doomsday and it's also not that. Like, no way. But I am not comfortable having some of these things to be out without the types of guardrails that we put in place. And it's very nice actually to see even ChemCrowd does have a function or a module, I don't remember exactly now, that is about safety and so on explicitly there. Similarly, I need to mention Cloud labs paradoxically become a very useful way of thinking about this because since everything is traced, the amount of metadata that you have is enormous, that you know the provenance of what's going on. And the folks at Emerald do have their own implementations for safety and checking and so on that give a good aspect of this. But I will say, Definitely not enough. Definitely not enough. Far from it. I really would like much more push from AI engineers out there trying to make this work from the big AI labs. And massive kudos to the UK AI Safety Institute that is trying to make this a little bit more formal.

Right.

So, and I know that some folks like to criticize Anthropic and Claude for being very careful, but like the reality is that like, yeah, we need to be careful. And many also about Google not right away releasing these things. Folks, like let's take a pause here and think about consequences, right? You, we want to push the technology as fast as possible. I want that. We all want that. We want the revolution. But, you know, don't always be so cynical. Think about that. There's a lot that happens that you may not know about.

That's a big part of why I do this show is I think that there's a strong reason, you know, on both sides of the analysis, you know, the opportunity to Accelerate science upstream of medicine, obviously incredible. It can't be denied. And yet at the same time, I do think the potential for things to go a little haywire is very, very real. There were a couple changes that I noticed from the original paper to the final one. One was that you dropped the use of the word emergence, which is Kind of been through the wringer. I'd be interested in your thoughts on emergence. And then another is that you dropped the section on generating ideas about new cancer drugs. And that I think is also super interesting. I'm curious as to why that got dropped. And I'm also curious as to how you would characterize We talked about this a little bit earlier when you were like, in the context of reaction optimization, it can match the Bayesian optimizer. But I'm wondering in terms of outside-the-box thinking, the kind of question where it's like, can you come up with a new cancer drug? There's something there that we're looking for these eureka moments. And I used to say we haven't seen any of them and now I feel like we've seen A few, but precious few. Emergence and eureka moments are my 2 questions. I specifically noticed those through the changes from the first to the Nature versions.

I think that the eureka moment that came here was really being able to do a lot of the tasks that otherwise would take a lot of time. We have this video. Unfortunately, it's not It's not public, but it could be comparing a very good researcher against a co-scientist on a given task. And co-scientist finished the task in about 4 minutes with code that is ready to be run. And this very good researcher, after about 15 minutes, had code that would not even run. So That I see as like, it's a, you know, a lot of people are like, okay, what did you discover? Well, what we discovered is what we are going to discover. Like, you know, like it's a different way of doing what we've been doing for centuries. That's what it is. First of all, the main reason why we decided to remove that section is because, you know, preprints are drafts. And that part ended up being a little bit out of focus with the overall story of what the paper is trying to explain and tell and inform. And we had to prioritize certain things. The development of, say, asking for cancer drugs or whatever, It was a little bit too open a question that in retrospect, I think that we— it can have a lot of very useful ideas. But I remember laughing pretty hard about like the agent at that point to just going for cannabinoids because there's so much about it on the internet, I guess. And like that kind of bias in GPT-4 that unfortunately leaks through in this, in this, not explicitly, but that, that was my, my feel at that point. And then, you know, this is on, on that. And we really wanted to do something that was much more focused. So it was like, okay, let's, let's remove this. And that's also a reason why I see the 2 works almost Different, like different works. So emergence. Yeah. So I am not a purist of that word. To say that I know that there are folks out there that believe that emergence is only something that you can talk about in the biological context, or that others come to say, like, there's no real emergence of whatever capabilities in LLMs and so on. I mean, come on, let's just use the word as— can we just use the word as a proxy for we didn't train focused on this and now we have these capabilities and so on. That's how I see it, right? And that comes, of course, from scaling and so on, different types of data. Different quality of data that goes into building these models. You see Phi-2, like it's a tiny model. It can do some really amazing things. Right. And are we going to say, oh, you know, Phi-2 does not have emergence of like, I don't know, like capability of solving pretty nice college-level physics problems? No, like it's fine. Sometimes I feel like we get caught up in certain words too much. Without actually trying to think about the overall message. And this was one of the words that I will say during the overall, you know, rounds and rounds of peer review and many online and not were pretty, for the lack of a better word, upset about it. So I don't care enough to fight for it. I think that It's much more important. It's like, hey, the work is this. It's a tool. It makes mistakes. I don't think we, like, no one trained for it to be a co-scientist and here we are. So is it an emergence capability? I'll let people decide.

Yeah. Exercise for the reader. Okay. So maybe last question. This is a good positive note to end on. You know, toward the beginning, obviously we talked about my experience as a research assistant and just how much grunt work it really was. Now we have co-scientists. Always worth reminding ourselves that this is the worst that this technology is ever going to be. So let's say I am considering a PhD in chemistry or chemical engineering now. What is my PhD going to look like as I start to have co-scientists as kind of the new I'm interested there in terms of how much faster can I go? Maybe also how much co-scientist budget do I need? And how much more can I accomplish in the course of a couple few years of research compared to the before? This could be your pitch for your lab or for the hard sciences in general, but I'd love to hear how you think this will be different now that we have these enabling tools.

Definitely a lot of things will change. We'll be able to pay less attention on certain mechanics that are time-consuming but not really pushing knowledge forward. So I cannot speak for PhD in cancer chemical engineering anywhere and everywhere. I can speak for the groups that I know that are using these things, like again, Philipp Schröller at PFL. Andrew White, and many others. I should really mention Alanas Puruguzik in Toronto, my postdoc advisor, that is one of the pioneers in machine learning for chemistry, period. And they are pushing extremely hard there for massive innovations. And the PhD experience will change, definitely. And I see, for example, I hope to see in biology become shorter so that people can go to the workforce faster because they don't, you know, though biology, there are other problems. Biology is hard because biology, but you get the idea. In chemistry, I, what I want from people that come to my group and work here is what are the big questions that you're trying to solve? What are, what like keeps you so, you know, focused on a problem? That you want to tackle, right? So there are people in my work, in my group working on things that are basically applied math, information theory, and there are people that are really pushing the boundaries in biocatalysis. And for all of them, what we have is now a way that dramatically diminishes the amount of time that they need to spend doing things that do not help them actually answer that question. And the questions can become more complex and more general. I am so excited about what we're going to be able to do going forward because we are not encumbered by having to worry so much about these things that took a lot of time but did not push, did not advance the knowledge. It's going to be awesome, right? It really is. We need to embrace it. And it's going to be Really great. And by the way, one thing that I really want to see, and I do not know how long it's going to take, is that, hey, this is not just chemistry or computer science or bioengineering anymore. Forget— let's just go over these boundaries because the main discoveries and solutions really occur at these interfaces between different fields. Right. That's how I really like to see things. And my group, we're interested in developing new reactions, for example, to tackle glycans and sugars that in the body, they are yet another piece of biology that has been severely underexplored because the chemistry of it is so difficult. Right. So we had Carolyn Bertozzi get the Nobel Prize in chemistry 2 years ago showing that you could do chemical, like, click chemistry. Okay, now let's go and follow her footsteps and explore those things by developing the technologies that will allow for us to modify sugars. That's the kind of thing that I'm super interested in. And I want, I just want people to really ask hard questions and be able to go into it fearlessly without needing to worry about Be it administrative things that don't really help them or writing code that is just an operation in the process. I think we are in for a revolution in how education works and how discoveries will be made much faster. We are already there, not because of Co-Scientists at all. Just look at AlphaFold. That was, you know, amazing. One of— definitely one of the most important discoveries of the last decade. Where are we going to go next? I don't know, but I know it's going to be fast and it's going to be powerful.

I think that is a great note to end on. You have given us a glimpse of the future with co-scientists at a minimum. Gabe Gomez, professor of chemistry and chemical engineering at Carnegie Mellon University. Thank you for being part of the cognitive revolution.

Nathan, thank you so much. This was awesome.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.