Automating Scientific Discovery, with Andrew White, Head of Science at Future House

You will never be able to reduce biology to like these cartoon diagrams. There is like complexity at every single level and it always plays a role. So I think it's a domain which is driven by observations and empirical measurements rather than a domain that's driven by like some sort of virtual in silico model that you can derive. I think sort of stops this, I don't know, ASI or AGI hypothesis that like a model that's so intelligent could just wake up one day and know how to cure cancer by just thinking through it. Sometimes it seems like we can solve the problems with empirical, like, machine learning models. Sometimes it looks like we can solve them with first principle methods. And I don't know, the only thing that does work 100% of the time is measuring it in the lab. And maybe all these methods are just approximations, and all they can do is increase your hit rate or decrease the number of experiments you have to do in a loop in the lab. Futurehouse is really Sam Rodriguez's brainchild, I think. Sam came up with this idea of a futurehouse. Which is basically like an FRO. Like, we want to be at the scale of that level, like $20 and $50 million on like a few-year timescale. But it's not like a 5-year goal. It's like a moonshot project that you may not accomplish in 5 years, or maybe you will know if you can accomplish it in 5 years, and then you'll need to go get more money to do it, or it'll be commercializable at that time.

Hello and welcome back to The Cognitive Revolution. Today I am excited to share my conversation with Andrew White. professor of chemical engineering at the University of Rochester, and now co-founder and head of science at Futurehouse, an Eric Schmidt-backed focused research organization that's building increasingly autonomous AI systems to accelerate scientific discovery. We begin by briefly discussing Andrew's background in statistical mechanics and molecular simulation, how his AI journey began during a 2019 sabbatical, how he came to write a textbook on deep learning for molecules and materials, And ultimately to his involvement with OpenAI's GPT-4 red team in 2022, which is where we had first crossed paths. From there, we unpack 2 of Future House's major recent releases, PaperQA and Aviary. PaperQA is a question answering framework that works across entire bodies of scientific literature using a mix of techniques including keyword expansion, full text search, contextual summarization, and large language model-powered relevance filtering to achieve superhuman performance on question answering, contradiction detection, and Wikipedia-style citation-supported topic summary writing. Here, Andrew emphasized Futurehouse's philosophy of optimizing for results rather than efficiency. They are willing to spend whatever compute or token budget is required and to wait for however many seconds are required to achieve the best possible output. This quality-first approach, as you'll hear, is one that I think a great many AI builders should take inspiration from. Aviary, meanwhile, Future House describes as a gymnasium framework for training language model agents on constructive tasks. In addition to creating conceptual clarity by distinguishing between agents, which contain core models and memory, and their environments, which provide tools and interfaces, this project introduces an interesting representation of agent systems as stochastic computation graphs. And very interestingly shows how agent systems can be trained end-to-end, even when black box commercial models are used at key nodes. I found Andrew to be so thoughtful in his responses, and he was sufficiently generous with his time, that I took the opportunity to ask a bunch of related questions along the way as well. One answer that continues to rattle around in my head was Andrew's argument that because better conceptual frameworks and automation platforms are quickly reducing the cost of real-world experimental work, Perhaps machine learning models' ability to run experiments in silico will ultimately prove less transformative than I had been expecting. There are, as you'll hear, a bunch more. And this was, believe it or not, Andrew's first ever podcast appearance. It took me months of friendly persistence to make it happen, but I think you'll agree that his skill as a scientific communicator is excellent, and he really should do a lot more of these going forward. If you're finding value in the show and want to help us spotlight more unassuming AI thought leaders, We'd appreciate it if you'd take a moment to share this episode with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Your feedback and suggestions, including for more pioneers of AI for science, are welcome too. You can contact us via our website, cognitiverevolution.ai, or feel free to DM me on your favorite social network. With that, I hope you enjoyed this deep dive into frontier applications of large language models for scientific research with Andrew White. of Future House. Andrew White, co-founder and head of science at Future House. Welcome to The Cognitive Revolution.

Happy to be here.

I'm really excited for this. This is one I've been begging and pestering you to do for a long time and honored that this is your first ever appearance on a podcast. So a lot to dig into, and I think it's going to be a lot of fun.

Awesome. Thanks.

So we first met actually on the GPT-4 Red Team, which has been now more than 2 years ago. And I usually don't do people's backstories too much because so many of them are, ChatGPT came out, I knew AI was going to be a big deal, especially in the entrepreneurship space these days. But being part of that GPT-4 red team clearly means you were on this earlier than most. And in looking into your background, I see a chemical engineering background. I don't see, at least on the level of LinkedIn, as much of an AI story. So I'd love to hear kind of your general What was it that helped you catch the wave early when you did? And how did you find yourself getting involved with the GPT-4 red team?

Yeah. So, I mean, it goes back for a while. So my, my training, my PhD and my postdoc is in a field called statistical mechanics. And statistical mechanics is like a branch of thermodynamics, but it's basically statistics of systems with extremely high degrees of freedom. So it's usually simulations of gases, liquids, proteins, things like that, or quantum systems. And so I've been working on this for a while. And then maybe in 2019, I got invited to UCLA for sabbatical at something called the Institute for Pure and Applied Mathematics. This is called IPAM. It's like an applied math group, and they wanted to do a topic on machine learning and physical sciences. So I went out there and I met with, you know, a lot of people were there. Yann LeCun was there. I think Yoshua Bengio stopped by for a talk. There was a guy named Pat Riley who has actually just started a new company, but he was leading part of Google Accelerated Science for a long time. did a lot of their fusion research, DNA-encoded libraries. It was a really great group of people. And we were all trying to figure out how these systems could be applied. At the time, it was pretty classical sort of machine learning. You do feature engineering and fit them. And there was one guy, what's his name? Matthias Rupp was there who had basically worked on this. Oh yeah, Anatole van Leeuwenfeld. Him and Matthias had built this system that was already state of the art for predicting energies of small molecules. And so people knew this kind of thing was going to take off in chemistry and physics as well. I think people were really excited about it. So then I didn't really understand the field that well. I went to the sabbatical and I went to a meeting in Tokyo actually on materials science. Sergey Kulinin was there, who's a big name in AI materials, and Lee Cronin was there, who's a big name in chemistry and origin of life and machine learning now. And it was a great meeting. I got to learn a lot about how these people were thinking about it. And Lee is a very controversial figure in chemistry, not because of anything he's done like wrong, but it's just basically these very strong opinions. And I think it was really cool to talk to him. He had some very strong opinions and Sergey did as well to learn about their thinking in the field. So when I came back from all this and this trip to Tokyo and the sabbatical, I ended up sitting down and writing a textbook called Deep Learning for Molecules and Materials. And it was kind of like course notes slash like textbook. I wanted to learn to use this new JupyterBook, which is like an executable book. So I wrote this out and it's been pretty popular since then, but I started teaching the class and learning more about it. And I started using language models pretty early. There was a paper by Gerben Ceder's group and Kristen Pearson, and they had done this work on like word2vec and material science. And that paper just blew my mind that you could basically embed the representation of a material using natural language, as opposed to trying to like come up with the right features. And so I started working in language and chemistry, and I worked with Glenn Hawkey, who's at NYU. And basically we were exploring, can you do things like drive VMD, which is a molecular dynamics visualization engine with voice recognition. And we were using, it was called Codex. Was that the original OpenAI language model? Was it Codex?

That was their code one, as I recall. They had GPT-2, 3. Codex was right in that timeframe.

Yeah, I think that's right. So I think that, I think they came out with DaVinci in maybe January or February, and then they came out with Codex a few months later. And so we were like driving MD, molecular dynamics simulations with this, which was really cool because VMD is written in this obscure language called TCL. I mean, I don't know. I think it's obscure. Maybe some people think it's bread and butter, but no one knew how to program in TCL. So every time we had to like write scripts for doing MD analysis, molecular dynamics analysis, we'd basically like Google a whole bunch, copy and paste code, like email friends. And so to have this language model that could just write the code a priori was super cool. And we wrote like a position paper. Basically, we wrote like a 2 or 3 page paper that was published in Digital Discovery, which is like a pretty cutting edge journal a few years ago. started by Ilana Sprague-Guzik, who's right now just a huge towering figure in AI materials. And we wrote this article about how we think the field could change with language models. And I put it out on Twitter. And at the time, I was starting to use Twitter more, and that was becoming a very exciting area. And some people at Opening Eye were trying to think about chemical, biological, radiological, nuclear safety. So, C-MORON is a word that people throw around a lot in AI safety. And not just AI safety, this is like a topic of terrorism and conventional warfare. So they were curious about seaborne risk and they sort of were looking at who was working on language models and chemistry. And there's a very short list of people. And they got my name and they got Glenn's name and they asked us to participate. And this is in August 2022.

Yeah.

August 2022. And so I started working on the red team there. And I mean, it was all like, nobody knew what they were doing. It was like completely new ground. Like, I remember like just trying random stuff. And it was a really exciting time. And I didn't know what I was doing really with language models and chemistry until a few, like, really, I think, groundbreaking papers came out, like this Miracle paper and the React paper. I think that really changed the perspective of what you could do. But originally, back in August and September, like, I was just, like, really confused on how these models could be useful in chemistry. But it was still, like, pretty cool to try a bunch of random ideas. And so that's how I kind of got into it. I'm happy to go more in depth. in this story, but that's sort of how I got to the red team point.

This is a little bit of a sidebar from our main topic, but I kind of came away from that red team experience, I would say, kind of alarmed. I was like, man, the pace at which these systems are getting better is really incredible. And I think now, looking back, I sort of infer that GPT-4 was more of a like maximalist moonshot attempt than I maybe understood it to be at the time. But nevertheless, the leap from 3 and even the Instruct series, all that stuff to 4 was just so eye-opening. And then I was like, but the control measures are not keeping up, or at least they did not seem to be keeping up at the time of that thing. And so I got a little freaked out there. Did you handle it with more equanimity than I did, or were you also a little freaked out?

I think I got A little bit spooked at first because it was the first time you could really see coherent, responsive answers. So I remember trying things like, you know, let's make nerve gas. Like, here's step 1, here's step 2. And I would ask it, how do you synthesize these things? And it looked really convincing at first. And it was like, really like, oh my gosh, like, this is going to change, like, how we think about deploying models. But then I started drawing out the chemical structures and like, looking to trace the atoms and how they move from step to step. And I realized that it was all like hallucinated, that it actually had no conception of how to move atoms between molecules. And so it was like a strange sort of thing where at first I thought that it was really technically brilliant. And then I realized it was actually like really good at bullshitting, I guess. And it's a strange thing because you like don't know if it's good or if it's bad. And the reasoning that it gives is actually quite plausible. And so for the first like one month, I was like, wow, this is crazy. And then for the next 2 months, like maybe October and then early November, I was like, there's nothing to this. Like it's just, it's going to be really cool. It's a nice incremental improvement. But then I started hooking up tools to the system. So I think in like late November, early December, I was starting to use some of the ideas that were like LangChain was average, was showing this stuff. I think it was around like ChatGPT API was released. And so people were showing that you could like sort of do these sort of React calls. And maybe that wasn't, no, I don't know if ChatGPT was available then, but like people were still with DaVinci at the time. And then I started hooking up to tools. And again, after that happened, I was like, oh my gosh, actually, this is maybe a big deal. And so that led to ChemCrow eventually. But I think, yeah, near the beginning of December, what was surprising to me is like, A, the mitigations that they were trying to build had no influence whatsoever once you started doing tool calling. And then B was that you could really make a lot of good progress if you sort of put the model on these rails of only using tools. As opposed to just sort of freeform doing chemistry by editing the molecule.

It's funny you talk about the hallucination in chemistry in particular. I did an experiment with it asking it to be my chemistry tutor, and I just kept getting this stoichiometry wrong and it could not, you know, when I got confused that it would get confused and then we were just off on a mess of confusion together. And I often think back on that in particular as a good indicator of How much progress there has been since then, even though, you know, we haven't yet seen the next 100x compute scale up. It's like these days it makes a pretty passable chemistry tutor and they've really ironed out a lot of those behavioral weaknesses to get it to actually do useful stuff in just obviously a lot more scenarios.

Yeah, I think what's surprising about these models is that they learn the fundamentals actually quite well. Like, if you ask it to reason through what reaction, what the rules are for this chemical reaction, or if you ask it to reason about this molecule's properties, that actually is quite good in general. What's strange is it misses things like counting the electrons and it misses things like counting the atoms and stoichiometry. And it's really, I think everyone is adapting to this thing, is that it learns backwards from how you would expect when having machines learn to do you know, STEM problems is that it seems to struggle with some of the more like robotic steps. And it seems to do quite well with some of the more freeform steps. And so it's one of those situations where you're going to see, I think, like a Grokking effect. Like once it can count the atoms and once it can count the electrons really well, then you're going to see a big unlock because it's not going to need any more additional reasoning things. So that's why I'm pretty optimistic about like things like O-1 and some of the more rumored, the rumored models that are coming up after that. I think, uh, they could actually sort of unlock chemistry to a point where you don't need to be on rails with these tools moving the molecules around.

How would you describe O1? Because you're still involved in the pre-deployment testing with OpenAI, right? So I imagine you did another kind of deep dive on, like, can this thing give an accurate nerve gas synthesis plan and whatnot?

Yeah, I would say that. So I was on the O1 technical report and yeah, this time around, I think Things have gotten a lot more systematic. So people have a much better idea of like where the benchmarks are. Still, for me, like, we decided to do some more exploratory stuff during the red teaming. So of course, there's already like benchmarks now. So I don't think like we need to be running the benchmarks. I mean, we put out LabBench from Future House, which is a benchmark of chemical, biological and laboratory protocols. And that was tested in '01. And so we were trying some really, I think, more Crazy ideas in the red teaming. And this time around, we got pretty far, but it still got hung up on like just some key steps. And so it's like a very strange experience because we, I think, had some pretty, pretty wild ideas and the models got really close to completing them, but it gets stuck on like basically one step in the process. And one thing that we didn't really get a chance to try this time around is consensus. So if you look at these evals, like if you run k equals 32, like you run it 32 times and take majority vote, that'll can actually sort of get over these little missteps. So I think, like, as the model that's available today, I don't think we're getting to that point, but I think we're very close. So I think the sort of next release is actually going to sort of allow us to do this synthesis planning, protocol planning. And I think OpenAI, like, addressed this a little bit. Their report, they showed that laboratory— it's called, I think, Protocol QA, which was from Future House— O1 exceeds human levels. And Protocol QA is like a model assessment of laboratory plans in the domain of biology. And so O-1 is basically above human level there. So I think that sort of shows you that it's getting to this ability to long-term plan and not make these kind of trivial mistakes that get it caught up. But I think in chemistry specifically, still not there. Biology, I think it's actually getting very close. Which is strange because usually biological sequences are very long and so like they don't fit in the context window. But a lot of the sort of ancillary things about biology, like how do you do cloning? What are the steps to get to this process? Those things, I think, are very close.

Hey, we'll continue our interview in a moment after a word from our sponsors. Does that reflect a sort of structure of biology? I'm kind of thinking of Dario's essay recently on what drives— I mean, of course, it's on the big picture of AI. Maybe we get to that in the big picture in a second. But his account of what drives progress in biology and medicine is basically that there's a small number of platform technologies that you sort of understand at a conceptual level and use on your problem versus— I was actually a chemistry undergrad myself, and my experience of chemistry was like, there's a tonne of different things and a tonne of different reactions and few sort of platform measurement technologies. Yes, but in terms of action steps, like not so many platform technologies, I guess, is how I would describe it.

Yeah, I think, I mean, Future House was built to be like a moonshot on automating science. And we picked biology for a couple reasons, similar to what Dario talked about. One of the reasons is that, yeah, it's a lot more platform-based. So there's like, let's say you want to design proteins, there is a standard way to make a protein. Like you can use cloning, or you can do cell-free protein synthesis, or you can make it on a machine. But basically, There's no stop to be like, oh, can we make this protein? Whereas you go to chemistry, every single molecule is very bespoke to make it. And there's a lot of questions about can you make it at scale? And it's not like a trivial step. So I think in some ways, biology is great, because it is kind of like a platform, like sequencing is almost free. And synthesis is quite cheap, you can basically make a 100 amino acid protein, you can order the gene to make that for like $25. So I think biology is a very interesting topic because, A, it's pretty clear how to test hypotheses and it's pretty cheap to test hypotheses. And B, I think what's unique about biology is that there is no limit to intellectual tasks. Basically, there's always going to be another organism genome to explore and annotate all the functions. There's always going to be some new protein that does something unusual. There's always going to be more dark metagenomic data where we don't know what this genetic material is. So in a sense, it's actually sort of an unlimited amount of complexity to explore or whatever method you want to use. Whereas if you go to something like physics, in physics, I mean, there are, I think, a lot of interesting topics in physics, but in some sense, it's pretty expensive to get data in physics. And it's also like there's not a ton of complexity to elucidate. A lot of time, it's a very reductionist thing in physics. You're trying to reduce it down to some equations or some relationships. Whereas in biology, there's just so many things to annotate. There is no, like, in biology, we kind of know the universal law of biology already. I mean, it's like evolution. And you go to physics and it's like trying to find the law. And so what's cool about biology is that you kind of already know the reductionist point of view and you're trying to like look at the more complex systems and understand how they work and how they fit together. So I think it's a very cool topic for those 2 reasons.

That's a really interesting comment that you already know the reductionist point of view. How likely do you think it is that there are major aspects of the full biological paradigm that we just have very little grasp on right now? I'm thinking, for example, of Michael Levin, who's been a past guest and has this idea that electrophysiological or electrobiosignals at sort of tissue levels are an interface that probably will be programmable. He's basically on a quest to kind of turn that into another one of these platform technologies. And yet he seems to be— of course, there are other people in that general space, but relative to the rest of biology, it seems like it's quite a niche. So I wonder how many of those other things you think, like what the sort of dark conceptual matter might still be out there.

Yeah. We were talking to Ed Boyden about optogenetics when he was saying, how did people discover? I think his lab was one of the pioneers there. How did you guys come up with this stuff? And basically said that we made what he calls a tiling tree. Is basically, you know, he wrote down the goal that he wanted and he wrote down like all the ways that they could try to attack it. And then they like sort of cut up space of ideas by like, okay, we could deliver something to the brain, or we could deliver a signal to the brain, or we could deliver, you know, via surgery, something physical to the brain. And it's like, okay, what kind of ways can we send information to tissue, right? Like we could try magnetic waves, we could try radio waves, we could try light that penetrates tissue, right? And so you sort of like go down these paths and it's, yeah, I think it's a field where I don't think we're going to run out of ideas ever. Like there's always interesting topics to explore. I think that's sort of the thing about biology is that you just have all these very complex systems with all these different interactions. And it's a very dense topic. Like I think I had an argument with somebody about like, how do you ever get to the bottom of like knowing a protein, right? People would say, well, once you have the crystal structure, you know, then it's like, okay, well, no, you need the statistical ensemble of the protein, right? And then it's like, okay, well, what about post-translational modification? Okay. Like what about things like the reactions in the protein, right? There can be these methylations or there can be like these, whatever, there can be these chemical modifications to a protein. And then there can be things like, okay, what if it has like a water wire that sends protons to the active site, right? You got to model that because the diffusivity of a proton by a water wire is different than the diffusivity of a proton by like hopping on like a hydronium ion. And it's basically just like, you will never be able to reduce biology to like, you know, like these cartoon diagrams. There is like complexity at every single level and it always plays a role. So I think it's a domain which is driven by observations and empirical measurements rather than a domain that's driven by like some sort of virtual in silico model that you can derive. And so I think it's going to be a domain which is, I think, sort of stops this, I don't know, ASI or AGI hypothesis that like a model that's so intelligent could just wake up one day and know how to cure cancer by just thinking Through it, it's really like a domain where you have to get out and you have to measure things repeatedly and get into this loop. And being super intelligent may not actually scale as much as just having a lot of good hypotheses that you can do. Anyway, so I don't know if that answers the question, but I was just—

Yeah, okay. There's a couple provocative ideas there that I want to come back to. But let's— you've mentioned Future House a couple times and a couple of the projects that you guys have put out. Let's just make sure we set the stage appropriately with What is Future House? How did you get hooked up with Eric Schmidt, who I understand is one of, if not your principal backer? And what's kind of the big audacious goal that you guys are chasing?

Yeah. So yeah, how do we start Future House? So I think, to be honest, Future House is really Sam Rodriguez's brainchild, I think. So Sam, who's the CEO, and him and I are the co-founders of Future House. Sam had this vision of like alternative mechanisms to do science. So Sam is the first one to sort of write down this idea of focused research organizations. And then he worked with Tom Kalil and Adam Marblestone, and they started Convergent Research, which basically has been launching these focused research organizations. And these are like 5-year nonprofits that are funded at like between $20 and $50 million that are built to just answer one very specific question that is not being funded in academia because it's like too big of a question, but not being funded in industry because it's like not a commercializable question. So like one example is making new model organisms or like understanding building the connectome for the brain or building technology to do the connectome for the brain. There's another one that's like doing Lean, like trying to build all the documentation and scaffolding for Lean so that it can take off as like the, you know, the language, the programming language of mathematics. But Sam sort of kept thinking down this path and he came up with this idea of a future house, which is basically like an FRO. Like we want to be at the scale of that level, like, you know, between $20 and $50 million and like a a few-year timescale, but where it doesn't, it's not like a 5-year goal. It's like a moonshot project that you may not accomplish in 5 years, or maybe you will know if you can accomplish it in 5 years, and then you'll need to go get more money to do it, or it'll be commercializable at that time. And so that's what Sam was sort of thinking about. I happened to be at this time, like, you know, we had finished ChemCrow, we had explored what can be possible with LLMs. We were exploring more stuff with combining like RAG. and tool calling and language models put together. And so, I told Sam, why don't you guys build it around AI automating science? So, Sam had been exploring ideas of new microscopy technologies, new ideas in sequencing, or new understanding the brain. But I said, you should focus on automating science. And so, Sam and I sort of brought— he brought, I think, the model, and I sort of brought what could be the topic. And we pitched it to Eric and he was very excited about the idea and he liked the topic. And Eric, the last few years, has been really excited about AI. So I think it really fit into his thinking of what the future holds. And so then we basically put together this organization from scratch. And I mean, there's been organizations like us, like ARC is one example, or Altos or Arcadia. These are all sort of non-traditional research organizations. I think the real difference between them and us is that we are focused, we are mission-focused, whereas I think ARC is maybe more of like traditional sort of PIs with their own ideas. We are focused on one mission. And one of the cool things about having a mission is that you can be a nonprofit. If you are a nonprofit and you don't have a mission, it gets very confusing about what you're trying to do. Like, how do you make decisions about what to buy, what personnel to hire, right? Because if you're not trying to make a profit, it's really unclear what you're trying to optimize for. We have a very clear mission. And it's an impossible, audacious mission. And so it's— we're never going to run out of things to do on this mission. And it helps us sort of guide the direction. So anyway, that's kind of like how the idea got started. There's been lots of like things we've done along the way, but we've sort of been around this nucleus of trying to automate science. And yeah, and then Eric has been the primary backer. But, you know, we are— we've raised funding from other sources. We've got grants from organizations. And so I think we're trying to go bigger in the next year or so in the scale of what we're trying Cool.

So I'll just read your mission statement. I copied it right off the website. Our 10-year mission is to build semi-autonomous AIs that can scale scientific research to accelerate the pace of discovery and to provide world-class access to cutting-edge scientific, medical, and engineering expertise. So that is pretty similar to what I hear when I read Dario's Machines of Love and Grace vision. The semi-autonomous is definitely an interesting call out there. In his vision, it seems like they're sort of on a 10-year timescale likely to become fully autonomous. He sort of envisions them being like the PIs and the humans, and maybe some other AIs are more like the helpers.

Yeah.

So in what way do you envision, even in the long term of AI, these things being Yeah, I guess there's 2 sort of wrinkles here.

I think one wrinkle is that I'm actually pretty bearish on laboratory robotics. You know, we may get there. But I think that there's been so many companies in biotech, especially, and I think in AI, that just tried to swallow, like, the mission of automating the physical world and just died because of it. And I think it's just like, you know, I don't want to say like Emerald Cloud Labs or Ginkgo Bioworks have died. They've made great progress. They've done great things. But this like commitment to automation, I think, is— it's a huge challenge. And so we can only solve so many grand challenges at Futurehouse. So we decided that we're not trying to automate the lab. And so I think one difference between sort of a fully autonomous, like, AI system that can do science is that it's going to have its own lab. It's going to have its own robots. It can do all the protocols, things there. We're not trying to solve that part of the scientific equation. If somebody solves Laboratory robotics tomorrow, great. We'll happily buy some, sign a partnership, whatever. But we're not trying to automate inside the lab. We do use standard lab automation. We have robot arms, we have liquid handlers, we have acoustic liquid handlers. We do all this sort of regular automation you'd see, but we don't have it as part of our mission because we don't want to get like stuck on that sort of an aspect. And then another thing I think that differentiates us is I don't think we're going to have a system in 10 years where we can just say like, explore the metagenome, and then we come back and a year and it will have annotated everything interesting in all of the genetic information on the planet. I think it's going to be more likely that we have a disease, a patient population, and we're like, what could be the biological mechanism for this disease? Okay, here's the biological mechanisms. Which one of these could we target with a biologic? Which one can we target with some molecule? What's a good starting point? So, I think it's going to be semi-autonomous in the sense that we're going to have a very clear quest and a very clear set of parameters. And we're going to more iterate with the system. And this gets back to this idea of biology is really an observational or empirical data-limited field, is that it's very unlikely we'll have some— you might have something in mathematics where you can just send it off to do math and you'll just be getting theorems or proofs. I don't know, once a quarter you get a dump of some. I think that's feasible in something like math where there is no empirical limitation. I think in biology, you're not going to just take some system that's going to say, do these 36 mouse measurements. You're not going to go order a mouse from a CRO and go do the experiment that way. It's always got to be, I think, done hand in hand with humans.

Just as a brief aside on these companies like Emerald Cloud Lab and Ginkgo Bioworks that you mentioned that are doing this lab automation, my understanding right now is that they have kind of like not exactly human in the loop, but sort of like human fulfilling some of the tasks. So they present to you a programmable interface, but behind the scenes there's a mix of robots and people running certain parts that are just—

Who told you that? That's supposed to be a secret. I don't know.

Yes. I think somebody might have said it on the podcast. I won't name any names because I don't want to get anybody in trouble, but I think the secret is at least somewhat out.

Yeah. I mean, I don't know. I guess this is the thing, is that some people call them biorobots or wet robots, which is when you have a human do the tasks that are really hard to And there's no reason to because it takes a human whatever two minutes. So yeah, I actually not I'm not really a purist in this sense. I don't actually care or know when something is you know no human allowed. I think there's like a term for like lights out automation. Like if you turn off all the lights in the building, is it still working? If so, then humans aren't in the loop. So yeah, you're right that Ginkgo and Emocloud Labs like they they'll automate whatever. They'll get to 98 percent and that's. as far as you want to go to get to the scale you want. I think there's other— what's it called? Medra, I think, is a company that's trying to build human gripper-style lab arms that move around the lab. And so then rather than having to build these bespoke instruments like that ECL and Ginkgo Bioworks build, it's like you go to a regular lab and you just put a bunch of robot arms and they move this stuff around. I think that's another path which can get there. Yeah, you're right. They are not 100% automated yet, and it's unlikely that they will be 100% automated for a long time because just getting that last sort of step is just a bunch of work for very little gain as far as throughput goes.

Hey, we'll continue our interview in a moment after a word from our sponsors. These are the human jobs of the future. Our comparative advantage is in manipulating some annoying thing in a huge value chain that we don't understand, but take just, you know, Chinese room style instructions of walk this thing over to this machine, make sure the gunk at the bottom is tapped out. It could be quite a funny vision. So, okay, that was one quick follow-up. The other one, in terms of just a kind of clarifying follow-up, was I get the distinction between go off and analyse biology versus we have one quest at the level of it can't just be purely internal thought. You can't O1 your way into knowing what actually happens in a biological system. I get that.

Yeah.

Is there another distinction there that you're trying to draw? Because you could still say, go off and explore biology and come back if it had tools, right? Or you had the ability to make these Emerald Cloud Lab API calls.

Yeah, I mean, you're right. You can imagine a world in which you have, I don't know, Science Dojo or something where you have a bunch of tools in it and one of the tools is go run a laboratory experiment. And maybe when you call that tool, it's dispatched to like a person who's got to go like figure out how to measure that. Right. And so you can get to the point where you're really trying to automate the scientific method there. And yeah, that's a great idea. I'm hoping we'll have more to share on that topic soon. But yeah, I think you're right. Like this is a completely valid path. I think whether that's semi-autonomous or autonomous, I think is like a minor point of contention. But I think that it may be possible that is something where you can just have very good tools. And you can slot in a machine in there and it can do well. One thing I always think about though is that, you know, we've been building these interfaces for these LLMs where you basically, you have, I don't know, like a big part of drug discovery wrapped up in a set of tools. You have a bunch of literature research wrapped up in a set of tools and you make everything on rails for these language models and you put them in, like you put O1 in there and say, okay, now use these tools to discover something new. I always wonder what if you put like a PhD student in there or like a gig worker that is on Mechanical Turk. Like, how good would humans do in this setting? Because it's like something we've never really tried before. Like, nobody's ever tried to make like a unified API for doing science and then put a human in the loop there. So I think we still have to see how good humans are at this task. I think it's a very interesting new setting where like you actually have reduced accomplishing science to a set of whatever, 25 tools that can be called programmatically. What if you put a person on the other end of it rather than a language model? Maybe this is like— the way of doing science, or maybe humans are bad at it and then the models will be bad at it and it's just not a way of doing science. So anyway, I don't know. I think you're right. That is a thing to explore. We are trying to explore that.

Yeah, that's quite interesting. I guess maybe before going into more depth on your specific projects, and we do have plenty of time and definitely want to get into those in technical detail because our audience loves the nitty-gritty stuff. One other thing that you had said that caught my ear and I wanted to make sure I understood was something to the effect of, for biology, you can't just run it in silico experiments and go from there to real understanding. My working mental model of how this stuff is going to work with ever better AlphaFolds and ESM3 and 4 and I want to ask you actually which of those things you're most excited about and kind of what you see as the most important trends there. But my working model has been you do the sort of experiment in silico and you measure the value of that by how often it is in fact predictive. Of course, you still have to go do the wet work to do the validation, but at some point you might be 10 or 100 times higher hit rate than We used to be because we can do these sort of, yes, this seems close. You almost need like a classifier on top. Is it worth actually going and doing the real-world experiment to validate? And it seems like there's potentially orders of magnitude improvement there in terms of just how much value we can get out of, say, existing throughput capacity, existing capacity for these sort of Real-world experiments. Is that your mental model too, or do you see it very differently than that?

Yeah, this is something I haven't really figured out for myself yet. Let me tell you about a few things in this domain that I think people miss. For example, right now you can do like very accurate free energy calculations to see if a small molecule will bind to a protein, and the cloud compute cost for that is like maybe $10 or $15. And the cost of making organic molecules and testing them against proteins has also come down. And so the cost of like the equivalent experiment is like maybe $20 or something. You could probably get it down to like $5, depends on the protein and, you know, what your catalog of molecules is. So right now you have this point where like actually both the cost of the chemistry and the biology has decreased at the same rate as the cost of the sort of calculations. Now you can Probably you would bet long-term on Moore's Law or on better machine learning models, and that might be true. So we may be getting near to a crossover point where we really do start replacing a big part of it with these in silico models. But if we get out, like if we zoom out a little bit, so like that's an example of like small molecule binding affinity. I think it's at the point where like they're close and they've both been dropping costs. We zoom out a little bit, like there was this idea like maybe 15 or 20 years ago that we could solve protein folding, a lot of biology by doing molecular dynamics. And there was this great sort of effort led by D.E. Shaw Research, where they basically put together like some of the smartest people, hardware engineers, like programmers, chemists, biologists. They put them all together in like Times Square, actually on top of a very nice building. And they tried to go down this path of like building the chips and the data center from like the atom all the way up to the macro scale to simulate proteins as fast as possible. And they simulated protein folding, you know, at, at, they were going like milliseconds of protein simulation time. And you can make these charts of basically, okay, like by 2030, we can simulate organelles. By like 2050, we can simulate whole cells. And then so by 2050, we can just simulate one atom per week. We can simulate the whole cell division cycle in an afternoon. And then we can basically run these virtual cell models. Only problem is molecular dynamics doesn't do chemical reactions. And it turns out that like a lot of biology is acid-base chemistry. A lot of biology is like actual like ATP, ADP, right? And NADH. Right. So you can't just model a cell by showing where all the atoms are and how they're moving. You actually have to model the chemical reactions. And so then people have built these reactive force fields, including machine learning reactive force fields. Maybe we can model empirically how these things work, not from first principles, but from empirical. And then you find out that actually the way that a proton moves through water is a quantum effect. You can't model that with just classical or machine learning. You have to actually— Have these like electron density calculations, and you actually have to make this like Born-Oppenheimer approximation to model the system. And that's actually important. Like biology works differently if you don't have this effect. And so you run into this like never, you never reach the floor of complexity when you try to model these systems. And it's like a really hard thing for me to figure out what the answer is to your question is that like, yes, AlphaFold has solved a lot of these things, but there's a big part of proteins that are intrinsically disordered. And yes, you can capture then maybe another 80% of those things with like Calvados, this force field from Crust and Lindorff-Larsen's group, or Martini, right? There's these things that can try to model coarse-grained proteins. Maybe you capture another 80%, but then there's some fraction of those that are not really valuable because whatever, you can't just use these coarse-grained models. So I don't know, there's like a lot of things that I've talked about here, but the point is that sometimes it seems like we can solve the problems with empirical, like, machine learning models. Sometimes it looks like we can solve them with first principle methods. And I don't know, the only thing that does work 100% of the time is measuring it in the lab. And maybe all these methods are just approximations and all they can do is increase your hit rate or decrease the number of experiments you have to do in a loop in the lab. So anyway, that's like a long-winded way of answering a question that I don't know.

Yeah, that's really very interesting. When you speak about the price of the actual experiments coming down. Are there caveats there around, like you said, for example, depends what your molecule catalogue looks like? What if I am trying to hypothesise a new smallish molecule that doesn't exist in nature, maybe never before synthesised, and I'm like, I'm looking for something that can do this thing. I'll generate a huge number of hypothetical candidates and then run that through A model, I can't synthesize those cheaply, right? Still today?

Yeah, I think this is like still an open question about what's the right approach. But there are these things called virtual catalogs that have reached, I think, I don't know, 80 billion molecules. So there's like 80 billion molecules that people are pretty sure they can make. And so what you can do is you can actually just work with those instead of a generative model and just basically screen these 80 billion molecules. I mean, I don't know what the number is right now. It's like somewhere between 50 and 100. There's something called ZINC. You can look it up. ZINC is like a database of hypothetical molecules, and it's in the dozens of billions. Probably, to be honest, like these things are made in a combinatorial, like, rule set. Like, these are all the things you can buy cheaply from petroleum side products. These are all the reactions that we know, and these are the ways you can combine them all. So you have this, like, combinatorial explosion of molecules. So at this point, there's pretty much unlimited number of molecules that you can order in a catalog. One caveat here is that all these molecules come with, like, a 80% success rate guarantee. That is, you can order it and 20% of the time they'll fail to fill your order, which is, well, it's fine. You can work around it. But then there is like a lot of groups that have basically built very good models for just making an arbitrary molecule. Like Philippe Schwaller, his group developed these like molecule transformers, which basically can predict the outcome of a reaction. And then you can combine that with some search method and basically predict how to synthesize arbitrary molecules. There are commercial products that can do this. IBM has one, there's a company called Postera that has one. And so basically frees people who design molecules from having to think too hard about synthesis. So I think in some sense, like the synthesis has been solved, but when you're late in a drug discovery program, like usually you're centered around some lead and then things get more expensive. You want to like modify it. And thanks to like internationalization, you can hire a chemist in China or India. And maybe the cost of one of these chemists is like $100,000 a year salary. And that includes the lab and all the reagents and things like that. So you can, and they can maybe make 20 molecules a week. So it's still, you know, it's not free, but it's reached the point where it's not too expensive. And so I think we do have a lot more freedom now in the chemistry we can design. But on the other hand, like the kind of targets that people are trying to hit, the kind of molecules people are trying to make have also grown in complexity with these new categories of drugs like PROTACs, molecular degraders, This induced proximity work where you're making these big fat molecules that have to accomplish like 2 or 3 tasks in the body. And so you're back at the same point you were like maybe 30 years ago where you have a whole bunch of complexity you're trying to pack into a small molecule. And so even though in theory we can make lots of molecules, you're trying to do so much in a limited amount of atoms that you're back to where you are and you need to be very careful and clever and you have only so many experiments. So anyway, I don't know. I actually don't remember the question was, but I just talked a whole lot.

Yeah, well, I think you answered it. I mean, it was just around my experience back as a chemistry undergrad was like small molecules were a PhD in many cases to synthesize. And there was not a, in general, some were easy, but many were hard. And there was not a, you know, there was not like an 80% success rate on just picking one out of a space of 100 billion.

Yeah, yeah, I do want to say that I think that is like reflective of a certain shift in mindset. So I think historically, maybe when you did your undergrad, like a lot of people would start with a natural product. So this is kind of the older idea of drug discovery, like phenotypical drug discovery, where you like grind up a bunch of dirt or frog goo or something.

It's always the Amazon frog. Yes.

Yeah, exactly. And you put that on a cell and you see, okay, does this like, does this cure cancer in my like model? And then what you usually go, okay, it cures cancer in my model, then we got to like find out what's in the frog goo. And then you like run it through the mass spec, and you run it through the NMR. And like, okay, now I know what the compounds in, now I need to start making these things and testing them. And this leads to like this natural product organic chemistry where you're trying to make pretty complicated molecules because they're not made from like room temperature chemical reactions, they're made from enzymes in some biological system. So those definitely not an 80% success rate. But I think a lot of the drug discovery community moved on from natural products, like rather than starting with these natural products, which are very complicated but often active molecules, they start with, I would say, more like petroleum-derived compounds. So these are things which are just part of like the global, you know, supply of chemistry, which is a lot of it is driven from like oil. And so because of that, the molecules that you see in drug discovery today are very different than what you would see like maybe 30, 40 years ago when they came from natural products. So the chemistry is usually easier because by default, from the bias of where we get them, they're coming from easier reactions, like a lot of, a lot of amine chemistry, a lot of simple basic building blocks, like maybe 75 basic chemical reactions that are done in med chem now. So the compounds look different. And some people argue that we've lost something by doing that. There's actually some nice effort lately, there's something called CoCoNut, where people try to put as many natural products together as possible and try to get people to train their generative chemistry models on those instead of the more petroleum-derived catalogs. Or there's companies like Octant Bio that are doing phenotypic drug discovery with these petroleum-derived ones, whereas most people are like, okay, I know I want to hit this protein, I'm going to hit it with these small molecules, design the molecule to fit the protein. Sri's company, Octant, they're like, okay, I want to fix the cell. And so they're working in the cell model, which is closer to what people used to do with natural products. So anyway, I think there's like lots of ideas in the field. And so that's why you see, I think, a change in the chemistry where it's become routine, because we sort of made this decision to double down on structure-based drug discovery with petroleum-derived compounds.

Yeah, fascinating. Okay, that's a really good update for me. I haven't stayed super close to the latest in chemistry.

Yeah.

And that's a good chapter for me to add to my understanding. One other just kind of conceptual question, you've got great answers to these, is what do you think about the multi-omic model approach? Evo for me earlier, I think it was earlier this year, was a real seemingly blockbuster moment where I was like, okay, it's going to happen. And by it, I mean sort of that the models will learn higher order representations of the space. And that's happened in language, but we've got lots of weird debates around, well, they learned all that stuff from us, so it doesn't really count or whatever. But in the biological realm, it's like there's probably ample stuff there that we don't know that if these models can figure out how to represent in an effective way, which Evo seems to be showing a little bit of preview of that maybe mechanistic interpretability techniques developed for machine learning become the backdoor into understanding what actually matters in, as it's hopefully learned from just a huge amount of multi-omic data. What's your feeling on that approach?

Yeah, gosh, this is a topic I think that maybe is older than evo- Brian's model.

Mm-hmm.

So I think if you go back a little bit to these companies like Recursion or Insitro, and to some extent Calico, although I don't actually understand what Calico does, but basically they try to build this foundation model of biology. You basically build some big matrix of small molecule drugs, maybe transcriptomics data of what genes are being turned on, what are turned off, and then take pictures of the cells, and that's the phenotype. And you can build this big model of how molecules affect the genes from the transcriptomics and the phenotype from the imaging. And then a big model can come out of that. And then you can sort of like the biology will just sort of fall out or the new targets will fall out. And this is sort of like a long— this is just like one step in the long promise of genetic information, right? Like there was the Human Genome Project, we could figure out all the genes. Then there was like, okay, we need to see how they change. And we get something like GWAS where it's like, okay, we're going to finally figure it out now. But GWAS didn't lead to this. Then we have these things like DepMap or something from the Broad where we're trying to understand lots of cells and And yeah, I think like the next one is like looking at the genetic— these genetic foundation models. I think that they can be very good for things like metagenomics, for understanding transcription factors, understanding regulation networks, but I don't think they're going to be like the sort of silver bullet that unlocks any deeper understanding of, you know, of diseases or any deeper understanding of what goes on inside of a cell. I think they're going to be one more model, one more tool in the toolbox, I don't see them as really moving the needle that much. So yeah, I don't know. I'm kind of pessimistic on this thing. It's like one of those fields where people have tried really cool sounding ideas and it should have worked eventually, but every single one has failed. And so now, like when you get old like me, that you are just cynical and everyone that comes out, you're like, well, we'll see. I don't know. You can read Derek Lowe. If you ever read Derek Lowe's blog, it's like the guy's, I'm not going to say he's cranky, but he's seen a lot of like waves of new ideas in biology, and it's pretty rare for things to work out. So, a cynical position is the default, I think, right now. But certainly, I think from just understanding biology, awesome. For curing diseases, I think we'll have to wait and see if it really moves the needle anyway.

Do you think that could be— because I think you could tell a very similar story about machine learning up until like 5 years ago, or maybe 10 years ago, depending on exactly when you want to start the clock. But There's like a sport now of like, everything was invented in the '80s. And I think to a large degree that is true. Most of the ideas that are now really working with sufficient scale, people had at least an initial take on a long time ago, and they just didn't have the scale of data or compute to get there. I don't know if you want to approach this from a handicapping perspective or But do you think that there is a chance that it's just a matter of all these things maybe were the right ideas, but we just haven't got quite enough scale to actually grok, so to speak?

Yeah. Yeah. I think you're onto something here is that one of the challenges of biology is the latency from like idea to feedback. Machine learning is great because you can try things and as compute has gotten good, you can see the results of these things. And in fact, you see a problem right now where basically people have cool ideas that work at like a 1 billion parameter model, or maybe a 500 million parameter model, but then you scale up to like a 10 billion parameter model or a 40 billion parameter model, that all of a sudden just doesn't work. Or if it's like you, it works at 500 million tokens, you go to a trillion tokens and it doesn't work. And so you see this thing of like, when you scale up, it costs more time and there's, you know, longer latency. And that is a great filter of ideas. And you find out, okay, actually the simplest ideas are the Or these ones are the right ones. In biology, we kind of have this as well, where basically you can do pretty quick experiments on a protein in a well, right? Like just a little protein you can do, you can manipulate it pretty quick. Then you put it in a cell and then, okay, it's a little bit longer cuz you gotta wait for the cells to grow and you gotta transform the cells somehow. Then you go to like tissue and then you go to model a mouse model, right? Like let's say I wanna cure aging. Well, unfortunately mice live a long time. And so you gotta wait for those mice to die or not. And so like you keep running these latency problems. And when it gets to drug discovery, it's at this latency where humans just suck at because it takes whatever, 7 years from mechanism to phase 2 trials. And it's phase 2 clinical trials tell you if the drug worked or not. And 7 years means all the people there are gone and moved on, right? 7 years means that people forgot why they were doing this in the first place. And so, you have these feedback loops that take so long. And that is really hard. So I agree, it could be something like we figured it out, but we won't know for 30 years because we need to try a few cycles to see if it works. Or maybe Recursion, maybe Insitro, maybe Calico, they all got it right. But Insitro just got their first platform asset, I think in phase 1 or something, or to IND. So it took 7 years for them to get from company conception to getting their own molecule into the clinic. You look at Recursion, Recursion, people love Recursion. Like, they do great social media, they have a big GPU cluster, everyone's excited about them, hope their stock price keeps going up. But Recursion, I think all of the assets that are in clinic right now from them, they bought from other people. So they haven't really got their platform going yet. So we won't know if their idea of building a big foundation model is the right answer for a while. And then things like Atomwise, a very early biotech doing convolutional neural networks on small molecules binding to proteins. Great idea. They had great work, great results, great people. But you know what happens is like some of these biotechs, they'll die because they picked the wrong target, not because they have the bad platform or their tech wasn't right. And so you just— really hard to separate cause and effect with these very long latencies, very expensive experiments, and so many variables going on. So I think it may be the same as machine learning. It might be some bitter lesson, but we're not going to learn it until 30 years from now or until we figure out a new way to do experiments, right? Maybe the human organoid research is going to get us to lower latencies on getting feedback from ideas. Maybe, you know, maybe RFK Jr. is going to come up with a whole new way of doing FDA approvals so that we'll get faster clinical trials. There's this like, there's this stupid thing in drug discovery. It's like the biggest, the most important bottleneck right now is like time to enroll the first patient in clinical trial. So if you wanted to cure diseases, cure cancer, if you wanted to save the world in drug discovery, you should be working on better like online ads for clinical trial site enrollment. And that should be where people invest the effort, but that's not a sexy thing to invest the effort in. But that is like, I don't know what it was, like 60 days or something is the time to enroll the first patient. And that 60 days is like very amenable to innovation, but people put their innovation in like finding a new hit from a machine learning model or like building a new foundation model to maybe improve your biological mechanism by a little bit.

If there's one thing I do have some hope for the Trump administration doing, it would be some sort of data liberation out of electronic health records to then have LLMs Crunch through and more proactively identify the possible patients as opposed to having to filter everything through the click of an ad and then you know collect the data manually.

Yeah, yeah. Actually, if you have a direct line to RFK Jr. or Donald Trump, or if you know, I'm a couple of weeks away, but it was the idea. Okay, here's something that you they can do: release all of the IND packets for every drug. That is like every drug which is clinically approved. Has filed an IND packet, which contains a ton of valuable toxicity, pharmacology, pharmacokinetic data. And all that data is no longer a competitive use for them because if it's already clinically approved, there's no competition. It's already under patent protection. It's already being sold. If they were to release all that data, there's a huge trove of data that we could use to better fit machine learning models. And it's very expensive data to get. And it's all kept at the FDA. And for, I think, No really good reason. So anyway, that is—

I heard there might be some of it in just like folders at Mar-a-Lago. So you could also head down there and take a left turn past the bathroom, and who knows what you might find. All right, I joke, but I shouldn't joke too much. I don't want to get on any retribution lists. Okay, so this is a great foundation. I think it's always really helpful to understand the worldview that motivates the work. Let's finally get into Future House. You can maybe give us a little bit of kind of— I think I have a growing intuition for why the AI scientists, and it sort of is like you don't really believe in any other silver bullet, and so we just need more scientists that can grind away at this for a long time. And obviously an AI scientist has certain strengths. I've particularly dug in on the paper QA and then the new aviary papers, but maybe give us kind of a brief history from inception to these most recent ones. And you can highlight any other works and kind of the overall strategy to tee those up.

Yeah. So I guess Future House started with ChemCrow. So ChemCrow was an early paper where we basically wanted to do a full scientific discovery process with language models and these tools. So I think the most important sort of problem we did in ChemCrow was we combined like retrosynthesis predictors, literature search, like code execution. We combined that with GPT-4 and we asked it to design like a new dye. So basically we said, okay, here's— I want you to make a molecule that's novel with 2 steps. It's like a 2-step chemical reaction. It's never been made before and have it absorb light at a specific wavelength. And we gave it to start with the supporting information from a paper And so it basically fit a machine learning model to the supporting information to predict the wavelength absorption of light in molecules. Then it basically used the retrosynthesis tools, did some searches to try to find a new molecule that would absorb at the specific light. And then it came up with a reaction procedure. And we had like a robot lab at IBM, IBM RoboRXN, which is like a cloud lab that we were able to use. And they were able to go through the synthesis procedure to make that new molecule. And then we were able to test it and it actually was very close to the wavelength of light. I think we were like 15 nanometers off or something. So it was like a great, like closed-loop discovery. There were some problems. Basically we were in a rush, we were in a time crunch. So the robot didn't finish that reaction. So we actually had a person go in and do the last step. So it didn't, it wasn't like a completely AI novel molecule, but we did other examples in the paper where it was all done with a robot. But I think it's like a full loop, right? Basically it does literature research, it plans out the protocol, it goes and executes it, and it does the measurement. But one of the things I realized is that scientific literature is such an insane concept to me. It's really surprising that we did this as a civilization for hundreds of years, that we built this big networked artifact of all of scientific progress. I can complain about how it's locked up behind publisher paywalls for hours, but I won't. I'll hold back. But basically, this sort of artifact is, to me, Whatever, like 99% of doing science is knowing literature. And the last 1% is like just moving things around a little bit to get to innovation. And so when we started Future House, we wanted to go after like the first thing is scientific literature. Like if you can sort of work with literature and you can understand literature and see what's been done and what hasn't been done, like see what's novel, what's not novel, that's going to be a huge fraction of the work of automating science. So that was the first sort of project we did was paper QA. So PaperQA, I actually wrote before Future House. I was giving a talk in Copenhagen in Denmark. I think it was in January or February or something like this. And I was in the hotel. It was rainy. It was like winter. So it wasn't much to do. So I just like was messing around with my laptop and made PaperQA. And it was like around these ideas of like RAG systems. And what was different about PaperQA was that what it does is it basically pulls up all the relevant papers. And then it, instead of like pulling the chunks of the papers and giving it to a final LLM to answer, which is sort of how RAG normally works with. What it does is it basically does a MapReduce, is it pulls up all of the chunks that could be relevant, and then it runs an LLM on each of these chunks to summarize them and re-rank them and then give them to the final language model to do the answer. And that solves like 2 problems. One problem is that language models are extremely sensitive to distracting information in their context. So when trying to work with scientific literature, if you got like a wrong paper in there or a section of a wrong paper, it would just blow up the system with incorrect information. So one example is like one of the questions you could ask it is like in the, I don't know, in the EBM trial, what was the size of the placebo arm? And it would pull up the wrong like trial methods. And if that was in there, it would like distract it from that information. So that was what this context summarization step did. So anyway, that was PaperQA and we spent like basically the concept was done very quickly and there was tons and tons of like defining what are correct answers, how do you work the literature, how do you make sure you get all the papers, how do you consider citations, journal quality, how well do humans do, can we make this repeatable, can we iron out bugs. And it took a long time. And sort of we capped off the project with something we call WikiCRO, which is once we beat human-level performance by making lots of changes— we have an engineering blog that walks through like all the sort of A/B decisions we made to get to beating humans— we wrote 20,000 Actually, I think it came out to like 18,000 Wikipedia articles that covered the function of every gene in the human genome, with the exception of a few, like 2,000 or 1,500, that have no papers ever written about them. So that basically was a summarization of all knowledge of the human genome, which didn't exist, doesn't exist until this was done, because Wikipedia only covered like 2,500 of them. So we added like 16,000 more articles to explain the rest of the human genome. I think that was sort of a good mark that showed that we can do this at scale. And we can do things like, with the scale that we can do it, we can answer literature questions as well as humans, and we can do it like 75 per minute. So we could like do what we call contradiction detection. So we have systems which can look for any contradictions or disagreement with any arbitrary statement from the literature, which is actually a really hard task, by the way, like to check all 250 million papers against An arbitrary claim. We can do that at scale of like every arXiv paper that's on arXiv per day. We can check every single one for any disagreement with literature and mark it. Or we can now like write the Wikipedia article for every disease that exists every 3 weeks, considering all new research that comes out. So I think PaperQA is like a really good, like, beginning to end. We beat humans, we deployed it, we built the infrastructure, we have an API, and we can run it at scale. And now that we have that, then it was like, okay, what comes next? And so we sort of took the lessons we learned in PaperQA, and that's something called Aviary. And what we did is we broke up PaperQA into 2 components. Basically, there's what we call the environment, and this environment is like the tools that are available, like look at citations, summarize literature— or sorry, summarize this paper, like do a Google Scholar search, do keyword search over some corpus. We turn that into environments, and then we have the agent, which is the thing that sort of drives the decision-making. That's like, okay, let's go do another Google Scholar search. Okay, let's actually search Semantic Scholar. Let's see who cited this paper. And Aviary is this idea that, okay, let's make a bunch of environments of scientific tasks beyond literature. So like ChemCrow, there's an environment like for ChemCrow, or we have an environment for designing proteins. We have an environment for like doing molecular cloning. So we have these environments here. And then what we do is we make agents and we've separated these 2 so we can try different agents, not just different LLMs, but we have agents that we've made ourselves with their own LLMs, or we have agents that have memory. We have agents that can do things like have multiple actions considered. They can do like reflection. They can do, you can have like a reward model baked inside of an LLM. So we've sort of tried to structure this agent interaction. And then what we do is when we have an environment and an agent and we train them together on some benchmark. And so, like we wrote, LabBench was a collection of benchmarks that we think are relevant for doing science. When you train them, you basically put them together and you call it a crow. And so, a crow, because basically it's like has tools, that's the environment. It uses language, that's the language model. Put this together, it's like a crow because crows are birds that can talk and also use tools. And then we are now deploying them and we built like an aviary, which is a bunch of different crows that can do different tasks and they all are connected. And we're slowly building like a platform of intellectual microservices, basically, go look for contradictions, go do literature research on this topic, design me a molecule that does this, design a plasmid which can clone this specific protein in this specific organism. And so putting all these things together, we were basically building the API for doing science.

That's fascinating.

Cool.

Really nice job laying out the progression and the vision there. Any number of follow-up questions, I guess. Going back to PaperQA, you're now on PaperQA 2, right?

That's right.

And the one thing that stood out most in terms of why it has worked well, because of course other people have tried to do this too, and I would say it's not easy to get above human performance in answering these questions. It sounds like one big Insight was process everything. I sometimes call this flash everything in honour of Gemini Flash because it's so cheap. I once set out the challenge for myself to spend a dollar a day on Gemini Flash, which is 13 million input tokens and a lot of information to process. So I actually have not achieved that. I've achieved it on a few days, but not on a consistent spend a dollar every day on Flash because I just Turns out it takes work to just identify those targets. But it sounds like that's one of the big things, was do this sort of filtering and identification of relevance with a language model as opposed to just with an embedding approach.

Yeah.

What other big drivers of kind of incremental progress would you highlight for people that might want to build a system?

Yeah, I think that was a big idea. Was doing that solves a lot of problems, like all of the effects of chunking, chunk size, quality of parsing. A lot of those things just disappear when you do this extra intermediate step. So it sounds like extra work, but actually it solves a lot of I think practical problems. It also increases the cost and time to get a response, and I think that's why you don't see like perplexity or elicit or anybody doing this kind of process. This like sort of two-step process because for a consumer-facing thing, you're. going from whatever, 15 seconds or 5 seconds to like a minute or 2 minutes. And I think that kind of reflects the philosophy is we just wanted to build the best possible system, forget cost, forget latency. So I think that's—

That in and of itself, just to highlight, I would say is a really important philosophy that I try to Johnny Appleseed my way around the world any chance I get. Engineers have way too much emphasis on Especially cost. Latency, I understand more if you have a user sitting there waiting. Human time is still precious, even if tokens are increasingly super abundant. But I always emphasize, yeah, deliver me the highest value thing first at any cost and at any latency, and then we can kind of work backward from there. So that's good. Keep going. I just want to hammer that.

Yeah, it's great. I mean, I think it's a philosophy people haven't really caught onto because people I think are still stuck in this mode of like, a Google search, you know, where it should come back fast and it should be like amortized cost where they spend a bunch of money building a big index and then they have low-cost queries. We didn't spend a bunch of time building a big index. We like do all the processing just in time. And so everything's more expensive, but, and we don't really amortize the cost. Another big effect is full-text search. Basically most search engines over research papers are not full text. And that just loses so much information. And that's why most benchmarks and most competing tools work with abstracts and titles, because those are accessible from search engines. It's very rare to get full text search. So we have built our own full text search. We open sourced like a version of that in the PaperQA 2 repo so that you can build your own index. We have our own internal, I mean, thing. It's just exactly what we think. So you can use like full text search in Postgres, or you can use Elasticsearch. And then Google Scholar is like just really, really, really good. And so we use a lot of like our performances from using things like Google Scholar and Semantic Scholar, which recently has full-text search. So that's a big change. I would say other things that are anything you can do to remove distracting information. And so a lot of our engineering blog is things like how to cut out potentially relevant things How to do summarization efficiently. We might consider 75 sources and then cut them down to 10. And that's better than considering 25 and cutting it down to 15 because you don't have as big of a cut. Yeah. So I think full-text search, so search, very, very important. And then this step, I think we call it in the paper RCS, retrieve and contextual summaries. Sorry, rank and contextual summaries. Those are the most important things.

One thing that did jump out at me as I was kind of perusing the code a little bit is it seemed like the control flow, like the actual agent itself, wasn't super complicated. The prompt didn't seem super insane. It was kind of like, here are your tools. It was sort of much closer to what I think people would sit down and write in their first attempt than you might think, given how well it worked. So I wonder if you had any reflections on that.

Yeah, I think the, what we have in the repo is like pretty general purpose. If you look at like, I think in the repo we have these configs and there's like a WikiCRO config or a ContraCRO config. Basically ContraCRO is like, just look for contradictions. And that's, I think, quite a different one. And WikiCRO is like summarize these things. I think we had a philosophy change. We used to have a very detailed prompt and we moved a lot of the complexity in the tool descriptions. So we have a something called, the package is called Aviary. And what it does is it will take like a Python signature with types and then the docstring and it'll turn it into like a tool for an LLM to use. And so the docstrings will end up as the tool description. And so some of the complexity is kind of hidden away in the docstrings where it's things like, okay, like try doing multiple searches with different keywords. And actually this is why Perplexity is quite good is their Perplexity's Pro Search. It does what's called query expansion is that you like, whatever, do your thing, your, the question you ask and it turns it into like keyword searches and this query expansion turns it to like 3 keyword searches. With different phrasings, and that really helps. Something we use as well is that we we don't use it. We tell the LLM like try doing multiple searches with different keywords. That's actually a really effective way to to get a to get a good good retrieval of sources. I'll mention another thing. We have this tool which we don't really advertise because we're not sure we can handle capacity, but it's called hasanyone.com. Have you seen that?

No.

Okay. So, we have this tool called hasanyone.com and it's a very narrow version of our internal tool we use for PaperQA. And it will basically do a search to see if anybody has done X. And it uses our paper database and our search tools and things like that. And yeah, it's a way to see because PaperQA 2 is like we changed some things for it to be more useful to people. Rather than using a full-text search you build and put on cloud infrastructure, it uses this Rust library called Tantivy, which will build the search index for you, like, based on per directory. I think it's much closer to what people want. We have our own full-text search that's, like, on our cloud infrastructure. So, you can use hasanyone.com to sort of get a sense of what we use internally.

Gotcha. Cool. Would these things work if you just did, like, a quick pivot to a totally different domain, like a social science?

Could you go to economics or education Yeah, yeah, it actually works well in many different domains. We've used it in lots of domains. We actually have a phone number we can text. So like, I'll be out at like, I don't know, a party or something like that. And I can text PaperQA and it will text me back like a response. And yeah, I've had— I've asked it questions all over the place, history, economics, whatever. It has access to all the papers in these domains. Like some of the— like we've concentrated on getting access to papers in like biology, you know, and medicine. But it has, you know, arXiv, ChemRxiv, medRxiv. It has like the ability to download open access papers. It has like anything we have in our cache. So it's able to cover a lot of different domains. But you will see this unevenness. Like if you go to like, I don't know, if you go to machine learning, it's great because everything's on arXiv. There's no real problems with open access, things like that. If you go to something like, I don't know, internal medicine, like New England Journal of Medicine, like they have some open access things, but they have the most annoying Cloudflare anti-bot stuff. So we're not able to access any open access papers in New England Journal of Medicine. And so it's kind of like field dependent, which journals are dominant, does impact the performance.

Yeah, that stuff is quite annoying. I mean, that has really been a big story of agents and general purpose web agents, whatever. It's like they're so often hung up by literally the dumbest stuff. It's a very weird phenomenon. One thing that I would say is not so dumb this is what I was going to ask before and finally remembered, was the challenge of just using figures effectively. I find, like, whatever, no need for false precision, but just on reflecting on my own mode of absorbing information from papers, a lot of it boils down to the figures.

Yeah.

And unfortunately, most of the time still, when you put a PDF of an academic paper into a system, it just kind of skips those and just maybe takes the caption, but doesn't really engage visually with the thing. Did you guys try to do anything about that, or is that something you're just kind of waiting for models maybe to get better at and hope to delegate in the future?

So we actually have a benchmark called FigQA in our lab bench paper. And FigQA is like an assessment of scientific figures where we took Pretty hard figures and pretty difficult questions. And actually, I think Sonnet, the October Sonnet model, beats humans on this benchmark now. So we've reached a point now where actually the models are definitely human or superhuman performance on looking at figures. So we are actually adding this now to PaperQA. I think, like, we're really bad at naming things. So like, we call it PaperQA 2, but like, we try to use SemVer, so like, to monitor backwards compatibility. So we're actually working on PaperQA 3. To version 6, and that version 6 is going to have the ability to look at figures. And what we do is just kind of like, you know, really simple is that we parse the table— sorry, we parse the paper as text for the purposes of search. And then we feed the images into the models for the later stages, either RCS or RCS and the generate answer, the final answer step. So we are looking at that now. I think the models have reached the point where they're good enough at looking at figures that you can actually do this at scale.

And so you chunk and then you just attach an image to the chunks and then at the very end you include the associated images?

Yeah. I think we haven't written a paper on this or done a lot of benchmarking on this, but the current philosophy is that the purpose of the text then is just for the search. And so the figure caption is there and then the search is there. And then what you can do is you just give the picture of the page. Including the text around it, as opposed to the you know as opposed to some parsing of it. And yeah, I think these models are so good at this that there won't be any issues. I mean, you can go down this path. We did this for building figuas. You can use label box, which is like human you know annotation to try to cut the figures out. There's a group thing called like SciInt or something. I think it's I forget where they're from UChicago or something, but they've written like a pipeline of PyMu PDF to extract the figures. But it's really fraught because PDFs are so different. You have a PDF from 1992 and the figure extraction's not gonna work, and then a paper from 2020, it might work. And so yeah, I think it's really difficult to extract figures programmatically. So I think just taking a render and taking an image of it is, is the way to go. Yeah.

Okay, cool. That's quite interesting. So no love for, not to say no love, that's too harsh, but no, no need in your estimation for Various tools? I'm thinking like Marker API. There's a number now that are sort of specifically about slicing up inputs like this.

So this is a good question. I think our philosophy, like going back to what we were talking about, is we just want to spend the most money and the most time, but for the lowest technical sophistication. So I think those kind of like these, there's a few of these companies now, and I think a lot of them are built around Detectron fine-tunes where you basically fine-tune a language model to parse up patents or research papers. And I think just long-term, what's going to win is just taking a picture of the page and giving it to a model which is sufficiently competent to actually look at the page and answer questions about it. And right now, if you just take a picture of the figure and you give it, including the caption and everything, if you give that to Sonnet, the October model, it will correctly answer questions about the figure. And so I think there is no additional need to go more complicated.

Yeah, I want to see, that makes me really interested in the, and I'm kind of surprised I haven't seen this yet, the latest Claude score on the ArcAGI prize. Because when they did the first 3.5 Sonnet, that was a major step up in coding, whatever, however many months ago. I took ArcAGI puzzles to it and I was really amazed by how poorly it could see, for lack of a more technical account. It could not count blocks or do the basic atomic functions that you would need to be able to do in order to do the reasoning later. This is probably another example of, or at least seems pretty likely an example of something that you kind of said at the top of, like, they learn in sort of a reverse order. Because I bet that if you could see more effectively, the reasoning would be a lot stronger. But such basic mistakes were made at the reasoning or at the visual level.

Yeah.

This new one, of course, can count pixels and click accurately on buttons. So where is their score, I wonder?

Yeah. I think when we built our benchmarks, the first thing we tried to do was, I would call, recall-oriented tasks. Like, here's a table, turn the table to JSON. Or, here's a figure, give me all the values of the figure. And the models are not very good at those recall-oriented tasks. But then we realized we don't actually care about that. We actually care about these precision tasks. Based on this figure, does the treatment of A improve response? Or based on this table, is method A better than method B? And it turns out that those precision-oriented questions are very good. Those are like, they get superhuman-level performance on them. But these recall-oriented tasks, like digitizing it or counting things, it doesn't do well at. And that's actually been like a weird kind of mindset shift for me is that— I mean, this is like, you go to Gemini, you put a whole 100,000 token document in and you say, okay, like what are the, what are the 50 main conclusions of this document? It'll do a shit job. But if you go like, what's conclusion 1? And it's this conclusion. What's conclusion 2? Okay, then here's conclusion 2. What's conclusion 3? And these models just work so much better in this like sort of, I don't know, turn-by-turn or autoregressive way where you like ask like single questions or single responses. Whereas if you try to get these like cohesive things where it's like summarize everything into JSON, they struggle a little bit more. And so I think in the figure understanding, if you ask it specific questions, it does well. But when you try to say, get all the information out of the figure, they don't do very well.

Yeah. Interesting. So what is the cost and latency to run a question? Maybe that varies to some degree by the scope of the question, but what do you roughly give in terms of guidance there? And is this something that you will productize in a full way? I looked at the GitHub repo. It has 6,400 stars, which is not a small number, but I also feel like the burden of, or not the burden, but the barrier of just having to figure it out is a lot compared to if you guys were to commercialize it. I bet you'd have a lot of interested customers.

Yeah, this is a good question. So to answer your first question, you can see the real latency of the full system if you go to hasanyone.com and ask a new question and it costs like Between maybe 15 cents and a dollar, depending on the question. Like if you ask, has anybody done X? And if X is like some moderately popular topic, but it's a niche thing, it will do lots and lots of searches and lots of exploration to see if anybody's done it. But if you ask like, I don't know, has anybody landed a UFO on the Washington Monument? It's not going to do a lot of searches. So it does depend on the question. But then like, yeah, as far as like the GitHub repo, we intentionally make it hard to set up. Because we really only want to be interacting with programmers, like hackers who know their way around these systems. I think if we have like a one-click setup or something really easy to set up, then we're going to have to like go down the path of supporting lots and lots of player people. And already, like, we run into so many problems with like, it doesn't work on Colab because Google will not do Python 3.11. Sorry, they won't do 3.10. I don't even remember at this point, but they basically are not updating Google Colab and we just use some Python 3 point, I think 3.11 features. And that's already like a big source of problems. And I just think if we make it super easy, it could be an issue. Although Ethan Mollick was able to get it to work on Windows, which was like a real testament to us being like, okay, we'll try to support Windows a little bit. So that was very exciting. But yeah, I think commercializing it is, you know, we get a lot of inbound, like people want to use it for doing diligence on acquisitions or diligence on investing in companies. People want to do it for like doing IP searches. People want to use it for their PhD. I will say that we generally let in academics, like PhD students, who will ask for access to the full API. And we did do a workshop where we gave people access to the API with rate limits of 5 questions per day, because we have this Python API that uses our server. And so, people will do these interesting projects where they have some idea-generating model, which then takes every good idea and then runs it through PaperQA to see if anybody's done it or if it's supported by the literature or if So there's like a lot of cool ideas that people have here, but we are always caught in this tension of like, we want to measure success as an organization as like number of novel discoveries. And does commercialization or revenue of PaperQA, does that lead to more discoveries? I don't know. It could if the right people are using it and the right like sort of progress is being made. Or maybe you can argue that like growing revenue of PaperQA is going to be like beneficial for the org. So we can hire more people. can hire more engineers or something. And so I think we haven't figured out that sort of like, that sort of balance between what is good for our mission and what is like just chasing a mirage or something, or what's just going down this commercialization path for no really good reason. So I think we haven't worked that out. But that is something we think about a lot.

Yeah. Okay, cool. So turning to Aviary, I just had a couple of in-the-weeds questions on that as well. It seems like there was a couple of novel things in this paper. One is the just kind of high-level concept work to try to get clarity on what is an agent versus what is the environment. There's been a lot of confusion there, or at least people are sort of doing things a bit differently. You were willing to put a stake in the ground, and it seems to me that the big thing is the memory is internal to the agent in your conception. I was a little less clear on, like, if somebody does it the other way, are you saying they're wrong? Or where does that matter? But yeah, I'm interested in kind of your philosophy of what is an agent.

Yeah. So how we broke it out in the paper is that I think a very working or very practical definition is that anything that you want to train is going to be in the agent. Anything that you think is untrained would go in the environment. They interact with language, with observations, and the actions or tool requests or function calls, basically. And so when we like go down this framing, it just simplifies a lot of things that we can try arbitrary agents, arbitrary trainers with agents, and then we can have arbitrary environments. And it's like a pretty opinionated interface, but it just sort of frees us up that we can really try to change things all these different directions. Moving memory from the environment to the agent, I think, is again one of these opinionated ideas. But when we thought about what is memory, like we thought about things like, okay, you can just append messages. But maybe you need to modify them once there's so many messages. You need to like compress them somehow, or maybe you don't want to do memory. You want to do like five previous messages. You want to use RAG for like the previous ones, right? And I think it's because of this is like so many ideas about memory. It's got to be a trainable thing. Like no matter how you write down memory, there's hyperparameters in there, like how you truncate it, how you compress it, what you include in previous messages. Maybe you want to cut out any images in previous messages or something. So I think all these kind of decisions made us realize that memory really is meant to be part of like a whole system. And then I think another decision we made is that we frame the agents as compute graphs, stochastic compute graphs. And this is again, like, I think an unorthodox— I shouldn't say unorthodox, but like, I don't think a solved problem because a lot of people view agents as these kind of state machines that are like somehow like going through multiple states. If you do it as a compute graph, it's like a forward to back, you know, from left to right. It's like a, you just shoot once. There's no recursion in it. And there's also no state in it. So the way that you actually get state here is that like, The input to the compute graph is the agent's previous state. So the agent basically takes in a previous state, an observation, and emits a new state and an action. And this framing is, I think, again, it's not how I think people naturally feel about agents. I don't think people think about them as compute graphs that much. They like to think about them as programs or some sort of stateful operation. But this just does a lot of things for us. It allows us to do backprop over the compute graph. It allows us to efficiently execute them. It allows us to do things like efficiently serialize and deserialize these things. as compute graphs without having to think too carefully about the state. So these kinds of decisions I think are very practical decisions, like how do you make progress on learning? And in the version of the paper I sent you, we didn't have any learning results in it, but we'll have learning results showing improvement, showing basically different training strategies having generalized learning across different environments and with different agents. And so I think the proof is going to be in the pudding. What can you get done with this framing? And I think we can get a lot done with this framing. And I think a lot of it is motivated by the fact that we just have to get past zero-shot. Like, I think a lot of agents you see today are all zero-shot. And in the zero-shot regime where you basically just work on the prompt, work on the hyperparameters manually, these opinions don't matter. Like, you can build a zero-shot agent in Python. You don't need framing. You don't need framework. And I think like some of the effort has been put things like on observability or traceability or like developer velocity. I think what ours is really built for is, can you train them? And so that I haven't seen as much in the field. Like, I think DSPy is the closest to what we've built. But in DSPY, I think they're not really in an environment, there's not really RL going on, whereas we really do online RL. We do online PPO with some of our models. Okay.

Yeah, that's quite helpful. I think the thing then that I want to understand better is this black box gradient estimation, which is once you've sort of taken an agent and said, okay, we're going to make it, you're going to want to iron out or uncycle the cyclic things or eliminate any structural barriers to making this thing end-to-end trainable. Now you still have a challenge of, well, what if I'm using Claude or O1 or whatever as part of that compute graph? And I tried, but I didn't develop a great intuition for what you are doing. The quote that I pulled out that I would love to understand is, to obtain these estimates, This is for describing black box gradient estimation. To obtain these estimates, we model the behavior of a language model and embedding nodes around the current configuration values with a multilayer perceptron and backpropagate through it. Yeah. Explain it like I'm maybe not 5, but maybe just like a lowly grad student or something.

Yeah. So this work came from Sid and Albert, and they're brilliant. So Albert actually, he did RL with Shani Diffrabiti. Which is a very good biotech company actually. And Gianni won the Unity RL Challenge, or I think he got second place in the Unity RL Challenge. These guys are RL geniuses. Albert, he also was, I think, the first author on PyTorch RL. So this guy's a genius. I actually don't think I fully understand what they did. But this paper by John Schulman, he wrote this paper on stochastic compute graphs, which basically lays out the framing of them and what the nodes are, the edges are, and talks about this. And we had built something like this and we found this paper. I'm like, okay, well, this paper actually describes what we built quite well. And he had written out these terms on how to backprop through stochastic nodes, which is not trivial to do. And then Albert and Sid, they basically decided to extend this to do it over black box nodes. And this is kind of like a no free lunch sort of thing where you really have to build a surrogate model, right? Because you can't backprop through this black box model because we don't know what happens if we change the temperature on the input versus the output because there's no way to trace the gradient through because we were calling Anthropic, we're calling OpenAI's API here. What they did is they tried to model the variance explicitly with an MLP. And so what you do is you basically, you go to do a call through the black box model. And what you do is you pick a temperature of 1, you call it whatever, 5 times with this temperature of 1. And that gives you like an estimate of the, I would call it like the aleatoric uncertainty. Like if I just call it over and over again, how much noise is there in that process? And then you get that out and then you can basically use that noise to measure the aleatoric component. And then the epistemic component would be like, what happens if I change these things? How much noise do I get? And then you can try to change these parameters a little bit and then get like an estimate of the change there. And then you can model this with an MLP. And the MLP is never going to be universal. Obviously, if the MLP could model the gradient everywhere, then you would have reproduced GPT-4, right? So what this MLP does is it gives you like an estimate of the local gradients by just watching how these input changes affect the output. It's very expensive because you're talking about running like whatever, 25 calls per input to estimate the gradient. So what you want to do is you want to basically do a rollout, just do forward inference using the agent a few times, and then you basically get a bunch of rollouts and then you want to go do one backprop step and you can basically choose which set do I want to do the most. Then you can estimate these gradients. You can try to do things like cluster them. So like, here's a bunch of similar outputs, so we'll use the same MLP for them. And then you can do this backprop. It's very cool to see in practice because what you can do, like in the paper we show, you can optimize the temperature for your model. You can optimize like the, I think it was the, the parameter in the prompt. You can optimize things like your, some hyperparameter upstream, like how many times you, how many periods you have in your prompt. A very unusual kind of way to do backprop. Now, in practice, really, this is more of a trick, I think, to show that it's possible. This does not lead to very good optimizers because basically, usually things upstream of the LLM are like, there's some optimum, like temperature 0.2 is optimum. And once you know that, there's not really much benefit in trying to change it. It's like very rare for there to be correlations. It's rare that you need to change temperature at the same time as changing compression factor of the memory. They're just not that correlated. So I think it's a really cool technical achievement and it's cool that it shows that you can do backprop through these black box models. But I think that as far as whether the outcomes change, the outcomes are not that important.

Interesting. Okay. So this would be really powerful if you're using an open model.

Yeah.

Not so powerful if you are using a closed model, though you can still get something out of it?

Yeah, I think the results we will have on the optimizers— what we do is we kind of split the models in 2. So we actually have a black box LLM, and then we have a second model which evaluates the outputs from the black box LLM. So with the black box LLM, generate a few outputs, and then we have a reward model or a Q-learning model technically that will then look at these outputs and choose the output that it thinks will be the best output. And so in that setting, you have a normal backprop through this model, which is an open or white box model. It's just a regular open source model that we're doing for the Q-learning. And then you have the black box model and you don't need to propagate to the black box model because things upstream of that are not that important. But anyway, this is, I think, the strategy that we've been using most successfully is this sort of hybrid Q-learning where you have an open source model as sort of the guide or the reward model for the closed source model, which is generating sort of the arguments to the actions. And the Q-learning one is sort of evaluating which one is the best given the situation based on past experience.

I don't know if this is maybe still in progress at the time we're talking, which is a little bit ahead of when the paper actually comes out, but how on some of these more frontier tasks, like, I don't know, you pick which one you want, but you've got high-end tasks. You're like, design me a new protein that's a little different from this old protein that'll have improved properties. How much improvement are you seeing with these sort of end-to-end trainable agents?

Yeah. So it depends on the task. Um, so we should see about maybe a 5 to 10 point improvement on the paper QA task. So answering literature research questions. I think in seqQA, we should see pretty high improvements, like maybe from 50 to 70. So like, sorry, 20 points, 50 to 70. And these kind of reflect how formulaic the tasks are in the environments. And so in the paper QA environment, we're basically trying to answer arbitrary questions and like basically take a multiple choice task by cheating, by looking at literature research. And yeah, the questions can obviously be like anything, right? So like the question can be like, consider these 5 papers and And tell me which one was different. So the actions that are chosen are quite hard. So it's not surprising that we don't really get a huge benefit. In the SeqQA, this is a molecular cloning task, we get a pretty large benefit, like this 20 points. And the reason why is the questions are quite formulaic. And so a Q-learning model will be able to outperform, or doing iterative expert improvement can improve it a lot because you're basically really doing a specific task with a set of actions. So I think it depends on, on the environment. I think this gets back to like something of, I don't know how useful these training procedures will be in the long run. I think right now we felt like helpless before we started this project. Like basically we can put in GPT-4o and we can do, make an environment and then, okay, it doesn't work. Okay, we try changing the prompt a little bit, throwing some in-context learning examples, maybe change temperature, but there's really no way to gather more data and get better. So having this environment at least gives us that ability to like see improvement from rollouts, improvement from gathering data. But I don't know if this is the right answer. Like, maybe this is just a stepping stone on the way to some better response. Maybe, you know, maybe Mistral will come out with an endpoint in a few months where you put up an environment and then they will train their models on it as long as you give them a reward model. So it's unclear to me if this is the right way, but at least we have sort of, you know, we've grabbed our destiny in our hands and can actually train models in these environments and make progress.

And so do I understand correctly that the main driver of improvement is— this would be consistent with your philosophy of spend what it takes to get good results— you are doing a lot of generations, and the reward model, or the sort of the Q models, you call it, essentially the one that is responsible for discriminating between those generations and determining which one to actually go with, that is the main driver of progress. Does that mean then that at runtime you have to do all the— you still have to do, given a new question, you have to do a bunch of generations and you're actually making a runtime choice?

Yeah. No, I mean, the Q-learning model is just an open source model we use. We've tried Phi, we've tried LLaMA. I don't know which one we'll eventually put into the paper, but that model basically is going to be tuned for this specific task. But we still measure on withheld questions. So if we go to— when we evaluate performance, we're going to new question types. And so it should be generalizable within the domain of using that environment for answering multiple-choice questions. Now, if we were to take that Q-learning model and then have the task is now to write a literature review, that model is probably not going to be great. And this comes to this question of how engineered do you want these agents and environments to be? And I don't know the answer to that question. I think it's an open question, but it's one of these things where you have to be able to write down like 150 tasks or 500 tasks and the model can get good at doing those tasks. But tasks that are not on your list of 500, it's unclear how well you'll do on them.

Yeah. Okay. So not to get too lost in the weeds, but just to be sure I understand the way the system works, given a new question at runtime, If I'm using a black box, if I'm using O or whatever to do my core function of trying to answer, do I have to do multiple generations and then have the Q1 pick the best in order?

Yeah, that's right. Usually these models, you can pass in how many completions you want. So you can do like K equals 8 or something like that. So yeah, it will generate 8 of them and then the Q-learning model will decide which one to actually emit to the environment. Because once you make a step in the environment, you can't go back. Although we have experimented with tree search. We actually have done tree search. This was a strategy to get ground truth or gold rollouts. Basically, we start with the black box model in this setup. Instead of having the Q-learning model pick one of the 8 outcomes, we try all 8, duplicate the environment 8 times. Obviously, you can't pick 8 because it's too high of a branching factor. You run out of space. But you can pick a branching factor of 2 or something, or maybe you can do some sort of beam search and you can get very deep. And we use that strategy to actually generate trajectories that had positive rewards because in these tasks, sometimes you don't get any positive rewards if you just start with nothing. And so we have used this, basically, I don't know, you call it Monte Carlo tree search. You basically go through these things by duplicating the environment until you get to a positive reward.

Yeah. Okay. Cool. I personally would love to see highly engineered systems have a big place in the future. I don't know if you've ever encountered Eric Drexler's Comprehensive AI Systems manuscript, but I do think there is something very attractive from a big picture safety and control standpoint to the idea that each of these things are kind of narrow. There's something in here that doesn't generalise well, and therefore it's kind of up to us to architect the overall AI supersonic system that's going to run a lot of society. But hopefully we'll have legibility on all these little subpieces.

Yeah.

I don't know if it goes that way, though. I mean, I kind of feel like O2 might just do it all, and then we're back to the big black box. What is your expectation for whether narrow has a place or Especially if you're willing to spend on the compute, if the great generalists rule the day.

You know, O2, I think it's going to be really hard to run it on every task in the lab. I think it's going to be something that is quite inaccessible to run at the scale we use things like GPT-4o or even GPT-4. So because of that, I think these kind of systems like we're building, where you can have some very good model generate a bunch of rollouts, that are ground truth or very good, or it can generate like a plan and then you have a simpler model execute the plan. I think that might be actually sort of what has to happen eventually. Because I think we are going to reach a point where like the compute, like the models, I think we've got lucky right now that the inference is actually the price that it is. I think we might start seeing this domain where basically there's very large expensive models that have a lot of test time compute, a lot of inference time compute. And because of that, they can be used to sort of help you train or sort of scaffold simpler models. So I think that may actually come to pass, especially with the rumors in the valley. I don't know how you say this, but with the rumors in the valley, I think that it's going to be unlikely we're seeing the same sort of test time inference properties for the next models.

Yeah. You look at those graphs of performance over inference time. And the inference time scale is logarithmic. So that is a nice, it's another one of these nice straight lines with the caveat that it's, you're going through a lot of orders of magnitude over the course of that graph.

Yeah.

And that is not going to be, I've been very impressed by how sort of egalitarian the developers have been when it comes to doing things like making GPT-4o free. And Claude 3.5 is also free in limited quantities for people to use. And just the fact that it's sort of genuinely like money is not really a barrier, hasn't been a barrier to getting access to the world's best AIs. There's been kind of an Andy Warhol, like, the president drinks Coke and the president uses ChatGPT, and you can use ChatGPT. It doesn't seem like that is about to break with this new paradigm.

Yeah.

And it's probably going to break both across people, but also 4 individuals or 4 teams, you're gonna have to like budget basically.

Yeah, I think you're right. I mean, and this is something we've experienced at Future House, right? Like with back to the conversation of PaperQA, right? Like PaperQA costs whatever, a dollar to use and takes whatever, 4 minutes or 3 minutes to do it. And that changes how you think about using these systems. And I think long term, that's why I'm still excited about Future House is that the kind of way that you can orchestrate these things where you can do like 1,000 things and each take 4 minutes and they each cost whatever, 50 cents, and you can pay $500 for this like giant intellectual task. That's just like a very different way of thinking about how we normally do things. And so I think the way that we automate science, it may not be the same kind of science that humans do, but we may be automating like a sort of different branch of science. So like some of the things we do here, like we run a combinatorial literature search. Let's look at like every antibody or every surface receptor against every disease ever reported in papers in this tissue, right? And we can just go and do that sort of combinatorial literature search. Then we get out like a matrix of findings, and then we can do like some, you know, light machine learning on those findings. And that kind of like task of like considering 10,000 papers or 20,000 papers at once is not something a human is doing right now. So we're not really displacing kind of human science. We're doing sort of a new category. So I do think it's going to require sort of a paradigm shift in how we think about these models because we're not going to get to this sort of like, I think we're going to get a little bit farther away from maybe like the advanced mode we know on like ChatGPT is like you're going to get far away from this sort of talking with a very low latency, very intelligent person into these sort of more like engineered systems that are very high throughput, very like very low or shallow intelligence distributed at scale, or maybe one very high intelligence on a few of these things and then you do this big MapReduce where you look at lots of different areas and do some simple intellectual task. Anyway, I don't know. I don't know what's going to happen. I'm very happy that we've got Avery out where we can actually train these systems complete. I think there's been good work on training models. There's been good work on training prompts, but to see like a complete system where you can go to arbitrary environments and have complex agents and be able to train them, I think is very exciting for us, but we'll see what the future holds.

Yeah, that's cool. Do you have any thoughts on kind of what you're looking out for next? Things like, I guess you could answer this in terms of what needles might move. I personally always look at novel hypothesis generation as one big thing where I think we really still haven't seen the tipping point quite hit. Mustafa from Previously Inflection, now Microsoft, was just talking about something that I do think would be maybe not so much for your use cases, but in general a huge game changer, which would be effective long-term memory where the thing doesn't have this sort of brittle context window, but some sort of next paradigm for memory. So just interested in general, what are the sort of around the corner sorts of things that you think will be the biggest difference makers if and when they do land?

I think like diversity in outputs. Have you seen AIDAN's benchmark before?

AIDAN?

Yeah. I don't think so. So this is like a benchmark, which is like basically they ask the model to generate like explanations for, I don't know, like what caused World War II. And it's like, oh, you generate 100 of these things and you'll find out that like when you, once you get past like 4, it's just the restatement with like different punctuation or different phrasing of the first 4. And there's this really a big difficulty in these models is like, you would think that you could do, you could generate 100 hypotheses and have them explore all of them simultaneously. But in fact, what happens is that they don't actually generate, you can't make them generate 100 novel hypotheses. Although you think you're doing it with like beam search or like top-K sampling, what actually happens is just changing the punctuation or the phrasing. And it's really only like 1 or 2 ideas. And so I do think we have a problem with these models that they really can't generate a lot of novel hypotheses, and they can't really split their thinking that well to different viewpoints. I think that holds us back in some setting because there's like sort of an upper limit to how many ideas you can explore simultaneously. Another problem is just like, you know, like what you said before, one thing I'm really watching, I don't know if there's a needle moving or if someone can build a needle for this, but how bots can interact with the internet. Like the internet has gotten like quite different than what it was like 10 years ago. You can't just like curl the web anymore. Most websites have bot anti-bot stuff turned on. Most websites like are hostile. Like Reddit used to be a great source of training. Stack Overflow used to be a great source of training. They're both now like hostile towards any sort of scraping effort. X changed to be closed. X used to be like my favorite platform. Twitter used to be my favorite platform because everything was written there was public, searchable, findable, citable, and it was like a public discourse. Now it's kind of indistinguishable from Facebook in some sense, that it's like closed off, walled garden, no bots allowed. Bluesky was very excited about, but they've already been overwhelmed by bots like posting there. And so I don't know what the future holds for them being like a savior of like a protocol which is accessible by both LLMs or AIs or bots and people. And so I think we're reaching this sort of, I don't know, like dead internet theory where, A, like the spots that are for people are being infiltrated by bots. And then B, people who want to make bots that just like do good things or do good work are like being stopped by other websites because of the— caught in the crossfire. So I don't know what happens next for that. But like we already have a problem where like open access papers are already in a pretty sore state. Like, we've been slowly getting better in science of getting things open access, but already open access papers are no longer accessible programmatically. And if you even do like license agreement or sorry, terms of service uses to download a paper, it's not accessible anymore, open access. And so I worry about like, if we need to build something different than the internet for like actually these systems interacting, or need to build something different than, than, you know, hosted PDFs, if we need to be more deliberate in how we think about building The world's model of the next generation of AI systems.

Claude just put out their all docs in one big text file, and that feels like the first half-tenth of a percent of a step in that direction.

Yeah. Yeah. I don't know. I keep thinking about this problem of like, yeah, like the Claude, like computer driver model. You can think about that. Or like, I think Molno was a great model of like, how do you point and click at things? And you could use that to drive things over the internet, but whatever, like 10 years ago, I'd be probably writing a Python or Perl script to just go to websites and download the stuff, and I don't know, use BeautifulSoup and just grab it. And there was no like crap now. But nowadays, you can't even get access to basic documentation now without being hit by these anti-bot things. And I don't know, like I would pay money. Like if I could have like a if I could have a quarter and give every website I want to scrape a quarter, and that offset the cost of egress, then great. I would totally pay that. But But man, it is getting really hard to do anything kind of novel and fun on the internet these days. It's like a big commitment if you really want to actually work on the internet with these systems.

Well, how about in closing, what is next for you guys at Future House? What are you looking for? You can give the pitch for any profiles that you would like to have reach out to you.

Yeah, I think Aviary is a big step for us. To have like a playbook to build these agents and environments. So I'm really excited for us putting all these pieces together. I think we're going to have some very cool demos of open-ended science once you have all these systems interacting. I think if anybody has ideas for PaperQA, we have this API. We work with people already on projects. We have some really cool stuff going there, but love to hear from other people with their ideas. Also, I think we're also open to people's ideas of strategy. There's been some great nonprofit tools that have been built, like Semantic Scholar came out of the Allen Institute for AI. And they actually struggle a little bit too about how to make a free tool available in an open source. CrossRef is a nonprofit. They sell their services for a small amount of money to companies. And so I don't think we've figured out this way of getting intellectual services out there in a nonprofit setting. And we want to, and I don't know how we deliver that. So I think we're still exploring that domain as well. So I don't know. Anyway, and we're also looking for great candidates, people who want to revolutionize the field, people who want to be automating science. We're always looking for them. And we're also looking for academic groups that want to try these things on, on their own ideas. Yeah, our goal is to deliver novel discoveries, and that can be from us doing it, or us like getting every scientist in the world using our tools to speed up literature search, or to speed up peer review, or to speed up designing molecular cloning Protocols.

Yeah. Cool. Offering those things as a service, I do think sounds like a, obviously a lot of infrastructure and a lot of maintenance love would have to go into that, but it does seem like it could be a massive unlock. So I don't know if there's that, it feels to me like there's something there, like that platform is one that I imagine a lot of people would be really excited to build on.

Yeah. Yeah.

Awesome.

Well, thanks for having me.

This has been worth the wait. Andrew White, co-founder and head of science at Future House. Thanks for being part of the Cognitive Revolution.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.