New in Nature: Google Agents Beat Human Doctors, Make Scientific Discoveries – With Vivek Natarajan and Anil Palepu

Hello and welcome back to The Cognitive Revolution. Today's episode features an eye-opening conversation with Vivek Natarajan and Anil Palepu from Google DeepMind. Their groundbreaking work on AMIE, the Articulate Medical Intelligence Explorer, and CoScientist represent what seems to me an important threshold moment in AI capabilities. I always say that if people truly understood what AI can already do today, many would be fundamentally rethinking their plans. And these projects provide perhaps the clearest evidence yet that AI systems are beginning to outperform highly intelligent humans in domains that require years of specialized training. Remarkably, this work was accomplished without special continued pre-training or extensive custom post-training that could only have been done within Google. On the contrary, these approaches could have been developed and can be replicated by Google's API customers, using commercially available models, advanced prompting techniques, and thoughtful agent design. We begin by discussing Amy. A year ago already, Vivek and co-authors showed that Amy was able to outperform human general practitioners in diagnostic accuracy. Now, with just a few important caveats remaining, Anil and team have demonstrated that it also beats human primary care physicians in analysis and treatment recommendations. The implications for healthcare access are obviously profound and are beginning to extend into specialized medicine too. The second Amy paper we cover shows that the AI system is already surpassing medical fellows in both cardiology and oncology and closing in on, but still falling a bit short of, attending-level performance. Notably, when cardiologists have access to Amy, their performance dramatically improves across almost every metric, suggesting a short to medium-term future in which AI doctors have the potential to both raise the floor for access to quality care globally, and also raise the reliability ceiling even for those of us fortunate enough to have access to first-world specialized care. This is, to put it plainly, crazy, and I am super excited that Google is moving Amy into something like a clinical trial in partnership with Beth Israel Deaconess Medical Center, a Harvard Medical School teaching hospital in Boston, for real-world validation. All that said, somehow, In Vivek and team's Co-Scientist paper, we see something equally, if not even more, amazing. This multi-agent AI scientist system, which is capable of accepting human input and feedback at any step in its process, was tested in fully autonomous mode on 3 increasingly complicated scientific challenges. First, drug repurposing, an advanced but reasonably well-defined task amenable to combinatorial analysis. Second, therapeutic target identification, a more open-ended challenge requiring the AI to understand and/or make quality hypotheses about causal relationships within cells. And third, and definitely most dauntingly, the wholly open-ended challenge of understanding the process by which bacteria achieve drug resistance. As you might have guessed, CoScientist, which by the way, Google is now making available to trusted partners, succeeded on all 3 of these tasks. And on the challenge of understanding drug resistance in particular, it blew everyone's minds by proposing the exact same mechanism that Google's independent scientific collaborators had recently discovered experimentally but had not yet published at the time of CoScientist's analysis. Overall, CoScientist demonstrates that AI systems are now capable of generating novel insights by connecting the dots between far-flung bits of hard-won human knowledge. This system is not simply regurgitating its training data. On the contrary, it's performing meaningful synthesis and proposing novel hypotheses that even human expert scientists recognize as both insightful and significant. If all that's not enough alpha for one episode, the implementation details behind these systems offer valuable lessons for AI engineers everywhere. First, structured reasoning proves far more effective than simple chain-of-thought approaches, especially when working with lots of input context. Both of these systems demonstrate the value of thinking carefully about exactly how you want your AI system to reason about specific types of problems. Second, finding ways to add new information or even just a bit of entropy, such as by giving the model access to search, is key to making self-critique and self-improvement schemes work over many rounds of successive iteration. And third, for now at least, the tournament-style evaluation process used to surface the best candidate hypotheses out of the many that were generated Seems to be an industry best practice that you can and should use in your own work. What's most amazing to me about all of this is that it was achieved before Gemini 2.5 Pro was available to use, meaning that everything we talk about today is still subject to a step-change improvement that should come more or less for free with a simple model upgrade. With this level of performance already established and core model progress continuing, The path to an AI doctor in your pocket and data centers full of AI geniuses is honestly becoming quite clear. AIs are no longer just tools for routine tasks. They are becoming legitimate thought partners in some of humanity's most complex intellectual endeavors, from diagnosing disease to expanding the very frontiers of scientific knowledge. As always, if you're finding value in the show, please take a moment to share it with friends, Write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. And please know that I sincerely value your feedback and suggestions. Whether we get to live in a post-scarcity society in which we all enjoy instant access to superhuman AI doctors, or perhaps on the other extreme, end up going extinct due to some crazy AI-driven scientific accident, seems to me to depend largely on how responsibly we handle the upcoming AI transition. And I take my role in AI discourse very seriously. If you think I can be doing better, please contact us via our website, cognitiveevolution.ai, or feel free to DM me on your favorite social network. For now, I hope you enjoy this conversation on the emergence of genuine AI expertise, which you almost certainly would have considered to be AGI just a few short years ago, and which I think you should still find absolutely mind-blowing today. With Vivek Natarajan and Anil Palepu from Google DeepMind.

Vivek Natarajan and Anil Palepu, authors of Co-Scientist and Amy from Google DeepMind. Welcome to The Cognitive Revolution.

Thanks for having us.

Likewise. Guys, again, so Vivek, this is your 4th time. What an unbelievable heater you and the team at Google have been on. I always say people, if they just had a little bit better sense of what is already out there today, they would be updating their plans in many ways that I just don't see people doing. So this is really an unbelievable example of that.

I came away—

there's 3 papers we're going to go down the rabbit hole on today. And I came away from this feeling like it might be time to call it. The AIs might in fact now be clearly smarter than me. And we can get into the nuances of what they're still missing a little bit. But I think almost everybody would read these reports on what you've been able to get the AIs to do and feel like they would have a very hard time matching that. And to get to this level that, like, a single AI model scaffolded in different ways and put to different purposes, it would be like years of undertaking for for me to get there for sure. So let's do some headlines. So there's 2 papers with Amy. We covered this once before. This is the— what is it?

The Articulate Medical Intelligence Explorer.

The Articulate Medical Intelligence Explorer. So I've been using a graph in some slides that I occasionally present for the last year or so since the first Amy paper came out, showing that basically When a patient chats to the AI, the AI is more accurate in its ability to diagnose the person than human primary care physicians are, as judged by other human doctors. So now we've got 2 new extensions to that. The first one is, and there are some caveats here that I think are definitely worth unpacking, I'll give you a chance to do that. It's basically now outperforming general practitioners, not just on the diagnosis part, but also on reasoning through what to do about it and ultimately recommending treatments, which is obviously a big part of what the doctor is meant to do for you. So, yeah, just unpack that headline for me a little bit. I mean, it's crazy that is out there in the world today. And again, I always say, like, I swear when I was a kid, if something like this happened, it would have been like headline news. Everybody would be talking about it. And there's just so much going on that some of this stuff even dramatic breakthrough as it is, doesn't seem to crack the consciousness. So tell us more about AMIE outperforming now, not just on diagnosis, but also on recommending treatments.

Yeah, I mean, prior to those, I think, first 2 kind of AMIE papers, a lot of the work in this space was on medical question answering. And there was some notion, right, that these language models do encode clinical information well, and they have a lot to offer. I think with those papers, we were trying to start to ask that question of like, okay, but that's not clinical practice, right? In clinical practice, the doctor is interacting with patients. They have to kind of gather this information themselves. They're not really presented with all the information upfront. And so that was the study, right? And it was doing this objective structured clinical examination, this format. And it was basically trying to see, can the doctor Or the AI in this case, interact with patients, gather that information, and still get to that diagnostic endpoint. And of course, I think we do a really good job in the paper, and I encourage people to read it, of describing the many limitations, right? This is a text-based chat. That's not how doctors talk to people. And kind of our future, our direction from there was really about starting to unpack some of these limitations. And so one of those limitations is this idea that You know, it's more than just, you know, from the first visit you see a patient diagnosing them. Like, there's a lot more to, you know, clinical care, right? And it's about managing a patient over, you know, multiple visits. You know, the endpoint at the first time you see a patient might be really like, we need to just order the right tests and we need to kind of set them in the right direction. It's not always that you know exactly what to do with the patient after seeing them one time. And so I think the management reasoning paper is really trying to unpack that. And we'll talk more about how that study was designed. I think it's super interesting. And there's other aspects to it as well. Rather than kind of more general recommendations, like, can we get really precise and can we kind of ground in accepted clinical practice guidelines? Can we ground in medication labels and start to turn these into slightly more actionable things? And similarly, with the specialty papers, it's trying to expand beyond kind of the bread and butter. common presentations of common diseases, like how does this work in more niche areas of medicine? And so I think, and we have a lot more work kind of just trying to expand on some of these limitations that we've identified in the first paper.

So how would you summarize the, I mean, if I understand correctly, it is still a chat-based interaction today, right? So one major possible extension would be to go to multimodal, but you guys have also done work on That separately, right? So maybe like, why is it not multimodal in this particular study? Was there a reason not to just like let people, you know, throw in selfies into the chat?

Yeah, I think it's more like when you're doing research, you want to isolate components that you're studying and do that well. And so when you add multimodal, I think artifacts add like necessarily more confounders over here. So we try to avoid that. And so it's just easier to study a text-based system to begin with. But clearly we've done work on multimodal before. I know that like on 2 podcasts we've spoken about like the Med-PaLM-M work and Med-Gemini. So all those components and pieces exist. And so very soon you'll see like the multimodal one also come out. So yeah.

Yeah.

Okay. The pace is relentless, that's for sure. So just to bottom line this one more time, basically we have With the caveat that it's not yet in this particular paper, although coming soon, multimodal. We have multivisit, kind of longer time horizon interactions between patients and doctors where the AI doctor is outperforming the human doctors on both the diagnosis and the reasoning through what to do and ultimately landing on sort of standard of care accepted proper treatment for these conditions. Anything else that we should— should I soften that at all, or is that like a good—

I mean, I think—

because that should be like on billboards, right?

I mean, yeah, I think the biggest thing, right, is, and we'll talk more about how we're trying to test this in the real world, but right, these are simulated consultations. They're patient actors, they're not real patients. And obviously there's a whole new set of challenges when You get real patients who, you know, all kinds of things can happen and they're not necessarily going to stick to their script. So I think that's a whole nother thing that we need to test and validate that our results truly do translate to the real-world setting. That being said, I mean, I think simulated consultations do show that, you know, we do have promise in this setting. And, uh, you know, we, we personally are very optimistic that these results would, would translate at least to, to a certain degree.

Yeah.

Okay.

So the next Amy headline is moving to specialized medicine, and there you look at cardiology and oncology. And here we basically, I would summarise the findings as the Amy system is surpassing fellows and closing in on, but not yet hitting the level of attending physicians in these specialist domains. So what further complications or caveats should we have to understand that?

Yeah, I think largely I'd agree with the notion that we have a lot of improvements in terms of being as consistent in all domains as the most experienced attendings, and there's room for improvement. I think the real headline for me, especially if you look at the cardiology paper, is that the types of errors they make are pretty different. AI and the general cardiologist. And I think the really exciting thing is we see that they're quite complementary. So the comparison we made was when we compared Amy to the general cardiologist head to head, there was some uncertainty about what was better. Some were better in some areas, some were better in others. But when we compared the general cardiologist with access to Amy's assessments to the general cardiologist alone, it was like like a landslide, right? In that case, in almost every aspect, it was considered superior when they had assistance. And I could talk a little bit more about maybe why I think that might be. Obviously, I think there's more investigation needed, but I think that is a really exciting aspect of this, that it seems to be just a helpful system in use by these experts.

And maybe to just contextualize that work a little bit more, I think if you look at the access to specialists in the country today, I believe for getting consultations on neurologists, for example, it's like 12 to 18 month wait time. And that's simply not sustainable.

Right.

And so the question is, okay, clearly we have better reasoning AI systems that seem to show promise in medicine. So why do we have like, how can we do better over here? Can we improve the status quo? And we should be able to do that radically. Like, no one should be waiting for 18 months to get like consultation on like a rather serious thing. Like, neurology is like not straightforward. And so, so that's kind of like the motivation. Like, there's like a lot of like access issues around specialist care, cost issues around specialist care. And so how can we do better? And that's like the key question that we're trying to address. And then I think the second thing is generally like if you look at like how medicine has evolved. I mean, that has led to these silos or these compartments and specializations. And so you have like this primary care, which is kind of like the front face to everything else. It's the door. And then you have all these silos. But the way that it has evolved, I think, is primarily because of like the limitation of the cognitive aspects of the human mind. Like there's only so much expertise that we can cram into our given brain. And so like we have to know. So because of those limitations, you have to go study neurology or cardiology or, I don't know, internal medicine, but not everything together. But AI systems don't need to have that kind of limitation. They should be able to do— I mean, given what we're seeing, they should be able to integrate knowledge from multiple different sources, multiple different disciplines. And so that's also like, there's a fundamental rethinking that is happening. Does the new age of AI-powered healthcare, does it need those silos? It could be possible that you not only have like a PCP in your pocket, but like an expert neurologist in your pocket. Obviously with caveats and things like that. So that's another class of question that we are trying to address over here. And then maybe the third point I'll just add on is this study is like, what, 3, 4, 5 months old now?

Yeah.

Yeah, so that was primarily done with the PaLM version of the models, if I'm not wrong.

The cardiology one was done with Flash.

I think Flash. Yeah, 1.5.

Yeah, yeah, 1.5.

Okay. Yeah, so it's 1.5, and so we've since then had 2 and 2.5. So yeah, I mean, in that study, obviously what we saw was that 1.5 was not as good as, say, attendings, but who knows with the new models.

Yeah, yeah, yeah. Um, yeah, that's an important caveat. Hey, we'll continue our interview in a moment after a word from our sponsors. Okay, well, let's do the headlines then for the CoScientist, because these are, I would say, similarly striking headlines. So CoScientist is basically, I guess I would describe it as an agent scaffolding type of setup where you— and we can get into the granular details, but we've seen different things like this before. I did an episode with James Zhao, who has kind of a similar thing with the AI Virtual Lab, and the original Co-Scientist that was made just within the first couple of weeks of GPT-4, which is crazy to think that they got anything out of that with like 8,000 tokens of context.

Yeah.

But even then, we're seeing some interesting stuff. But this I would describe as just kind of taking sort of all of the lessons learned about how to make agents work from the last 2 years and hitting the gas on all of them, and then coming up with a system that basically can do science, right? I mean, it's not executing the actual physical experiments at this point, but I was really amazed by the different things that you tested the system on. The first one is kind of like 3 levels of challenge of the problem that the system is given. The first one is like a relatively well-scoped thing that you could kind of grind through, that might be the one that I might be able to do with some real effort. And that was drug repurposing, right? So take a drug that's out there, look for other things that it might be useful for, and you have kind of a combinatorial approach that's available to you. So if you set that up and go through it systematically and you have decent judgment around each individual sub-question that you ask, you can sort of imagine how an AI system would be able to do something like that. But then you go up a level to the second scope of task, which is identifying new therapeutic targets within a particular kind of diseased cell. And this is starting to get at what I've often called one of the grand challenges in biology, which is just like, what causes what? Obviously, we know that there's super complicated causal graphs going on in the cell of this promotes this but inhibits that and yada, yada, yada. And we don't— I don't know if you guys would venture a number, but I've kind of understood broadly that we have a long way to go in terms of really mapping that out. Maybe we understand 10% of what the graph is in today's world. So the challenge here is to basically go into go into a cell, so to speak, and try to figure out, can we identify something that we can target with a drug that will actually make things better? Obviously, this is hard given the vast complexity and the many unknowns in cells. So, okay, that was just the middle one. Then the third one is, can you figure out why or how Bacteria are becoming drug resistant. There was like a little bit of a hint, I think, because there was, there was one sort of observation, right? That there was something that was conserved across a couple different species that sort of was, was a notable observation that, you know, sort of served as a seed to unpack the challenge. But beyond that, go figure it out, right? Super open-ended, really, really tough. And that is like a daunting question for me to consider. And really would have you kind of spinning your wheels, I would think, in just an absolutely vast literature for a long time before you would even, at least for me, before I would even have any sense that I might be able to start to contribute to the discussion. Okay, those 3 problems. Bottom line is AI is able to do all 3 of them pretty well. And in that last case, it actually surfaced as its number one candidate idea for the mechanism of the drug resistance, something that had been discovered experimentally but was not yet published, right? So you guys partnered with an academic group that was doing this research and they had the answer, but nobody else had the answer. It's not in the literature. And the system was able to grind through all this content that it had available, the whole kind of vast body of medical and biological literature. And I mean, crazy, right? Landed on the exact number one hypothesis that turned out to be the actual answer. I was really blown away by just how successful it was. So tell me more, I guess, in terms of what else— are there any caveats that I glossed over, or are there any other eureka moments that you would want to highlight from those results?

I think the last one was kind of interesting and also funny because I don't think José and Tiago, who are collaborators at Imperial, would take offense at me saying this, because I don't think they really believed that AI could do this thing. So I think we've been trying to— we said, okay, we have the system, José and Tiago, do you want to try it? And then it took us a few months before we got enough time with them. And then I think after like enough pestering, they were like, okay, we have some experimental results in the lab. We'll just try and challenge your AI system. Let's see what your AI system can do. And so, yeah, I think it was roughly around like Thanksgiving. And so they sent us this prompt and I think that's detailed in another preprint along, which was co-timed with the CoScientist paper. And as you said, I think there were some clues in there, but like it was not totally giving it away. And so we take that prompt, we set out this system to it and it runs for like a couple of days. It spits something out and then like Yuro, our, like the first author on that paper, and he's an amazing like technical fellow. He's probably the least well-known technical genius at Google in some ways because he likes to keep a low profile. But yeah, so he just sent it over to them and then— I think it was Thanksgiving. It was like late in the evening. And so Yuro and Alan were both based in Europe. They were like offline. And then like within 10 minutes, like Jose was also based in London. He sends us like an email and He was like, "I need to talk to you right now," and I was like, "Okay." I didn't understand the seriousness of it, but I was like, "Okay." I mean, I'm not doing anything better. We can talk. And then he was like, "Vivek, are you like reading my email?" I was like, "I'm not sure." I was like, "What are you asking?" He was like, "No, no, no. It seems like you're reading your email." And then I was like, "We do many things at Google, but." Reading your email is not one of them. And then he goes into it. He was like, I've not published this anywhere, but your AI system came up with the same set of results that we hypothesized and we found in our experiments. So I'm really surprised. And so he was like, okay, do you guys get responses from ChatGPT? I was like, no, no, no, that's not possible. That doesn't happen. And so he was like, okay, so if you're not reading my emails and then if you're not— if you don't have any information from ChatGPT, then It's likely that you have something like really, really magical. And that was kind of his response. And then he said like, yeah, the first one is great, but we also like sent him 4 more. And he was like, all the other 4 also make a lot of sense. And I think immediately after the Thanksgiving break, he was like, okay, I'm gonna like set a few of my postdocs to work on this. And they've been like working on validating those other hypotheses. And so it was kind of that moment, like where someone who's like very pragmatic, very experienced. has spent like a decade, in fact, several decades on the field, like when they have these kinds of reactions. So that was kind of like, okay, that told us like, okay, we might be onto something over here. But again, I think like it was not like one single moment, but it was like, okay, we— it's a very hard thing to do, right? I mean, to get like AI to not just like synthesize and integrate and summarize information, but help traverse this tree of knowledge and like uncover new original things and knowledge and facts about the world. And to do that reliably, that's a super hard problem. In some ways, that's like the holy grail of AI. And to think that, okay, like a system which is relatively simple in nature, like we'll probably talk about this more, it's like, it's probably the simplest version of a system that you can imagine. And like throwing a bunch of compute at it, you're already seeing evidence of that, like you're seeing evidence of this happening reliably, that felt like, okay, super magical to us.

Yeah, it's crazy. So this has been something that people have been discussing recently online a bit. Like, Dwarkesh has sort of advanced the idea that why don't we see AIs coming up with these? They have this incredible breadth of knowledge. Shouldn't we be seeing more connections made, more sort of insights across this super diverse knowledge base. His contention is like, if I could have all that knowledge, surely I would come up with more insights than the AIs currently seem to. And I think he even kind of went as far as to say, like, I haven't seen a single example of this. And I feel like some of the things that I've covered definitely seem like they could be said to count, but there's always a lot of details and caveats and sort of eye of the beholder type of stuff. But this seems pretty clear to me that the fact that this was essentially independently discovered by human scientists in a lab and an AI system in a data centre over a couple of days in parallel, to the point where the scientist accused you of reading his email, presumably not super seriously, but nevertheless, Is there anything, is there any reason that we should sort of not take this as a genuine discovery, a sort of qualitative eureka moment from an AI system?

Yeah, this is not the first evidence. I think right in 2023, right after we did our MetPaLM, and that's where actually the genesis of this CoScientist work is actually. Right after we published Med-PaLM, there was this professor from Stanford, Dr. Gary Peltz, who reached out to us and he was like— and I think you probably remember Tao, who came to one of these previous episodes before. He called both of us up and then he was like, Vivek and Tao, you don't know me, but your AI system can potentially help millions of people with rare diseases. I was like, okay, Gary, that's a nice introduction. Please go on. And then he was like, I know that your models are trained on a lot of scientific literature. And I think they can help me discover useful facts about genetic diseases. And so, I mean, that felt interesting enough for us. I mean, if we were able to help him, probably we could help a lot of people. And so we started working with him and on this problem of genetic discovery. So can language models, can they come up with the right kind of causative factors that are responsible for a given combination of phenotypes or symptoms? And so we started doing that with Med-PaLM. And even with Med-PaLM and later with Med-Gemini, we saw that these systems were able to do this. In fact, with Med-PaLM, we— like, Garry was working on like mouse models and like a very specific kind of hearing loss for which he had this NIH grant back then. I don't think he'll ever get that kind of a grant again, but it's a separate discussion. But so yeah, he had that grant. And so one of the hypotheses that the model came up with, it was like a biogenic model for hearing loss, which Garry had not thought of before. And so he went ahead and did these CRISPR knock-in experiments in his lab. And he was able to reverse the course of the disease. And so we had written that up and it's still under review in actually a prestigious venue. And then later on, we used our more advanced version of our LLM, which is MedGemini, and we extended that work to human variants of unknown significance. And even there, based on retrospective data, we see these systems are able to do pretty interesting work in genetic discovery, which is like one of— it can be cast as a hypothesis generation problem. But the key thing is, at that point of time, when we were using these LLMs in like a pretty much like a crude single-shot fashion, it was very unreliable in this act of hypothesis generation. So literally, for that Med-PaLM 2 to come up with like one hypothesis that was very, very helpful, it had come up with like thousands of things that were utter garbage. But I mean, we were very grateful to like work with Gary who had the expertise to like very quickly discard away things that were nonsense and also had the patience to like work through all of them because could easily imagine another scientist or maybe someone inexperienced who would like say, looking at the first 5, oh, this is utter garbage, it's not working. And then we would not have like pursued this line of work at all. And so it's about like getting together and working with the right people who believe in this, right? And so that like, you know, put us on this journey, like, okay, how do we make this more reliable? Like, we should not be sampling like thousands of times to get something useful. Rather, every single generation should be something useful and the system should be like well calibrated over here. And so that kind of like led us to this. And so the simplest way to do this would be to like, you know, call like an LLM repeatedly and hope that it leads to something useful. But that very quickly fails and it leads to like these degenerated solutions which has this mode collapse. And so It became important to introduce net new knowledge into the world and also think about how you can gamify that process in some ways and introduce repeated new helpful feedback that can help the system self-improve. And so that kind of led us to the design that we eventually had. And in hindsight, it should be quite obvious. I think it very naturally follows how the scientific method works. Or if you ask a scientist how do they come up with new ideas, you'll see they will just roughly compartmentalize into the set of agents that we had. They're like just doing the same thing. But it was just like this iterative process of like, okay, we tried something, it didn't work. Okay, we tried to improve it. And then we eventually ended up with this design, which is like, I think, remarkably intuitive.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, how do you— there's a couple of like, maybe quasi-philosophical questions that come to mind there. One is, how do you think about the relationship between hallucinations on the one hand and creativity or hypothesis generation on the other hand, it seems like we have very— people have very different intuitions about that. And I'm also not quite sure myself, like, do I— when we do sort of extensive post-training to try to minimise hallucinations, does that help with hypothesis generation? Because maybe it makes the models more disciplined in their reasoning? Or in some ways, does it hurt because they're maybe less willing to come up with a very random idea that every once in a while it's these random ideas that are the big ideas? Right. So how do you guys think about the— maybe that's a false trade-off, but yeah, how do you think about the relationship between those behaviours?

Yeah. And Kirsch, hear your take.

Yeah, I mean, I think intuitively there's some aspect of hallucination does foster creativity in some ways. It's interpolating between the data it's seen, right? And it's kind of necessary for hypothesis generation to deviate from the script in a bit. But yeah, I mean, I don't have a I don't know if I have a clear understanding or answer to that, but intuitively at least, that's how I feel.

Yeah. I mean, I always used to think that hallucination and creativity are like two sides of the same coin in some ways. And I don't think this no longer holds true, but what we used to see with the previous generation of models was it was actually much more helpful to use the models without post-training for this task because they were then most likely to come up with these crazy ideas, right? But now I don't think that's any longer the case because we've been able to like systemize— we have been able to like put a structure around that process of like coming up with new ideas. And so it feels like now the process is much more reliable, but maybe we are sacrificing, say, some like crazy new things that would require maybe like an unnerfed, like non-post-trained model to come up with. So we don't know. So I think we've gained like reliability in the process, like in terms of like consistently coming up with like new original thoughts. But like, we don't know if you're like sacrificing something else over here.

Yeah. And I think one advantage of our system too is like we allow like diversity of models, you know, so, you know, we can kind of get the best of both worlds. And through this tournament process, through this ranking, like we have a diverse set of hypotheses and those can be reranked and hopefully we see high quality hypotheses kind of bubble up to the top.

Yeah, I mean, I think maybe it's both is kind of what I take away from the hallucination reflections there. It's like if you're in the context where there's a rich literature and it's more about really working your way through it, then maybe you want a disciplined reasoner. And if you're doing something where there really isn't much to go on, maybe you want a sort of more whimsical hallucinator. And as you said, maybe get the best of both worlds with some of these setups. So let's describe the different setups. I mean, there's multiple different systems here that they have their sort of intricacies, but it seems like at a high level, If I understand correctly, what you're doing to design these systems is basically just kind of introspecting or maybe interviewing people and sort of saying— one of my favourite questions in AI automation in general is, how do you think about it? So it seems like you're kind of doing that and saying, like, okay, how do you think about it? What do you do next? Then once you do that, what do you do? And you're basically turning all these steps of a process, whether it's the scientific method or it's the diagnostic process or the reasoning to treatment process, you're basically just kind of mapping that out with a subject matter expert, yourself or somebody else, and then creating little subagents that are sort of prompted to do those subtasks and then kind of scaffold them together and Also some interesting details on giving them tools. Literature search is obviously a huge one. I was also interested to see that in the one project, the model did have access to AlphaGo and maybe some other things. So you're increasingly giving it something like the full complement of tools that a human could use.

Yes.

And then it seems like after that, it's like, turn all the hyperparameters up, kind of do more rounds, more generations, more more evaluations, more rounds of feedback. And I guess my impression is that if you do that and you have the budget for it in today's world, you could probably be successful at almost anything. But maybe tell me if that's wrong. Are there things about the designs of these systems that you think are like, actually very sort of important hinges where if you designed it a little different, it wouldn't work? How sort of within that general framework that I outlined could you go a bunch of different directions, or do you feel like it actually is sort of a narrow design space that actually works?

Yeah, I think maybe there's a high-level narrative change that's happening over here. So Yeah, this is my 4th time on the podcast and I think previously we discussed Med-PaLM, Med-PaLM-M, and Med-Gemini. And the key with all of them were all like med. Like we were taking some generalized model and then we were trying to fine-tune them and specialize them. I think the key differentiation with the current version of Amy and CoScientist is we're no longer trying to do that fine-tuning step or that specialization step. And part of it is because, like, okay, some of the data that went into that fine-tuning and specialization step is now upstreamed and part of Gemini. But it, yeah, I mean, it just feels like a better approach to set things up is by simply having agents with specialized prompts and chain it up together in a nice manner so that, yeah, it just gives a lot of flexibility and control. And so it does away with this need for fine-tuning and specialization. I'm not sure what you think about that.

Yeah, I kind of agree. Not that there's no role for post-training, but Certainly in our first AMIE paper and Med-PaLM, all these papers, it was really about this medical data we're curating or creating and how we're making that. And that's what was driving the model's success. And in these latest papers, that's really, I think, taken a backseat to simply how we're designing the system maybe to perform these sort of tasks at inference time. And I would say I don't think anything is particularly hyper-optimized. Like, you know, there's probably certain ways that, you know, someone could design a system that does this stuff better. I'm sure, I'm sure there is. I think like our goals have been kind of largely to build to where it's like a functional prototype for these and, and kind of get there quickly and do the kind of tasks that we're interested in doing in each, in each study. So I think there are, there's so much room for improvement in all of these.

How many rounds of iteration did you go through as you built the system? And were there any moments when it was like, oh, we tweaked this prompt, or we switched this, we put this agent before this agent and kind of rescaffolded it that way, and all of a sudden you saw a big leap?

Or—

yeah, so tell me about those.

I mean, for the Amy work, I think we rely really heavily on auto evaluations. So I think we do try a lot of different prompting, a lot of different configurations, but ultimately it's only finite how many things you can try before you're like, this seems good, this is good enough. So I say we use that as a rough signal. We also honestly rely pretty heavily on vibe checks and just— And I think you can pretty quickly start to see differences. I think with our manager reasoning agent, for example, We saw a big difference when we started drafting concurrent plans and then refining them together. We saw a big difference when we did this top-down.

Can you unpack that just a little bit more? What was the before and what was the after?

So before we were just generating one plan and basically now we're generating 4 different plans. They might have some similarities, some differences, but we found basically in a sort of self-consistency style manner, right? That the model was able to combine these plans in a way that it kind of was able to take the good stuff from each plan and leave out the bad. And that's just something like once we tried it, it was like our internal auto-evaluation signal was very clear. This was making a huge difference. But in terms of hyperparameters, it's like we went with 4 plans. We could have gone with 8, we could have gone with 2. I don't think we optimized every hyperparameter in that sense, but I think the big things we were able to pick up. from this kind of signal.

Yeah, I think on the core scientist side, it was a much longer iteration. It was almost like close to an 18-month project in some ways. And it was driven by this need to make this process of hypothesis generation more reliable. And so, yeah, it was like, okay, we decomposed these tasks, we tried to get individual models to work on them. And I think the good thing is, again, like, what it has shown was how good individual LLMs are getting at instruction following. And that also does away with the need to fine-tune and specialize and whatever, because simply, if a model has the knowledge and if you give it a precise set of instructions, it can just follow it. And so that just makes it much, much easier to create agents and chain them up together, where these agents are specialized to do specific tasks but are just prompted versions of the general purpose models. And so, yeah, I mean, like, it was a process of like, okay, setting up some evals, seeing how well the system does, where the weaknesses are, and then like, again, iteratively going in and fixing those weaknesses by adding like specialized versions of agents or like fixing the prompts. And then ultimately, like, there was a process of like, okay, let's just simplify the architecture. We don't need all of these things. And so, yeah, that ultimately led to this design. So I would say, yeah, it was a lot of iteration, like figuring out where the weaknesses are. Trying to cover those weaknesses. And then at the end, like, okay, let's do a simplification push.

Yeah, I've been through that on the less world-changing project myself multiple times where you're like, a new model can do wonders for the simplification of your system. Yeah, I've definitely lived that. So, I guess, you know, in terms of what is driving the The improvements, you know, 18 months ago, we maybe like couldn't— I mean, it's all happening so fast, right? Like, I even have some, you know, lingering sympathy for the stochastic parrot people because it's like, as of GPT-2, you know, that was probably still mostly true. At this point, you know, it's pretty clearly not. But 18 months ago, you know, you maybe couldn't models just couldn't do certain things that now they can. Seems like just core model progress is the tide that is lifting all boats dramatically. Is there anything else that you think is just super important, or is it really just down to foundation models getting better and just grinding out the process of figuring out how to use them?

Yeah, again, I think on the Co-scientist side, I feel like long context was an important part of it as well. Because we don't have an explicit memory store in the system, but the fact that Gemini models instantiated can take up to like 2 million tokens, it means that we can just like, you know, generate ideas, generate reviews of them, run debates of these ideas, and like generate these walls of text essentially, which can just become feedback. And this just put all of them back into the context of the model in the next round. And the model just figures out how to make sense of it and use the feedback in a very implicit manner to improve. If you did not have these long-context abilities, like the ability to reason reliably over millions of tokens, then you would not be able to do that. You would have to engineer RAG-based systems and maybe you have to end-to-end train them and that would have been a lot more complex. But all that has been made remarkably simple just by these long-context abilities of Gemini, which I think is a little bit underappreciated in the field because we don't have enough of these I don't know, Studio Ghibli-style viral moments with long context, but it just enables a lot of these practical applications. I think that's the same with—

Yeah, with the management reasoning, it's like, yeah, I mean, like our recall for that paper, right? Like the management agent, its whole point is it's trying to take in clinical guidelines and reason over those to produce management plans. And yeah, I think if we were solely reliant on always picking out the right guidelines, we would struggle, right? with the long context, we don't really need to worry about it, right? We take in a bunch of guidelines and with 256K, right, we're going to catch something relevant regardless of whether our retrieval system would have been able to do that. So yeah, I definitely agree on the long context. I think just like with Code Scientist in particular, we're talking about a really long timescale. We have this thing running for days, right? I think just like that inference compute increase is offering a lot of benefit as well.

I'm just looking up— a friend sent me just in the last day or two Fiction Livebench for long context deep comprehension, one of obviously many benchmarks that look at these things. And the Gemini 2.5, even relative to Gemini 2, is absolutely just crushing on its command of long context. And I've definitely felt that in my initial testing of it. This does feel like something that's hard to go viral. It's hard to go viral with a notion of, like, I had hundreds of thousands or a million tokens of context that's all very idiosyncratic to whatever I'm working on. Hard for people to even know what you're talking about, right? When you post that on Twitter, the contrast between those context window and the length of a tweet is pretty severe.

Yeah.

But I mean, it's amazing that was working that well, because just looking at these benchmark results, it's like 2.5 stands out in a massive way relative to everything that had come before. Did you, I guess, qualitatively or vibes or your own sort of sense? I would have said up until 2.5, I would have felt like, I'm not so sure if I can just dump Hundreds of thousands of tokens in and like, yes, it can handle it, but does it really handle it? Does it really have command? And especially if it's like material that I don't have full command of myself, it can be very hard to evaluate that. So I guess, how did you handle that and how did you know if it was actually making effective use of the super long context?

Yeah. I mean, in the AMIE setting, I think, like I mentioned before, we really relied on auto-evaluation. And I have to shout out Valentin, my team member on the paper, did a great job setting this up. But essentially, we also didn't know how well would this work if we stuff a bunch of guidelines into context. And we found initially we were going to go all the way up to 1 million. We found that actually we seem to be getting better performance when we drop it down to 256K. Ultimately, there was a clear difference between the presence of that much knowledge versus trying to do it zero-shot or with one guideline or some retrieval or something. So we were also uncertain, but we tested it internally at least, and that seemed to work.

Yeah, I think on the co-scientist side, it might be a little bit more unscientific. I think what we relied on was primarily the redundancy aspect. Because like, okay, you generate some ideas, you review them, there's a tournament that happens, you get like a bunch of feedback from the tournament. And then you, when you put that back into the state of the system, like you're not generating only once, you're like generating again, like numerous ideas. And so your hope is because of that redundancy that creates, like at least one of the generations would catch the key elements of the feedback that has been generated and propagated back over here into the system. So yeah, I wouldn't say we have like any specific evals that target measuring how well the long context is doing, but rather by engineering in this redundancy, we're hoping that it would be effective. And yeah, and then the other distinguishing factor for us in this work was at least that I feel like there's a lot of like science assistant style, like scientific discovery style projects, not just, I think in a lot of different places, but where they do get a little bit hung up is in like curating these really nice cozy benchmarks where you can hill climb on. And that was not our philosophy at all. For us, we feel like the key deal is like if a system does something useful, we should rather sprint straight ahead so that we can validate it in the lab and then hopefully take that onward towards a real meaningful discovery. And so that was what we were most focused on. If we engineer a system, we go straight to the scientist who's an expert in the field, we show them the idea, and if they like it, we try to convince them to validate it. And if they validate it, it becomes a discovery, then yeah, great. So there was that thing where we really wanted to not micro-optimize on specific benchmarks and hill climb on them and wait for— I think if we did that, because there's so many different components in the system, I think this work would have taken easily another year if we tried to make it the best. Rather, we were just focused on, let's test them all up together, let's get it to do something reasonable, and then we'll do end-to-end validation.

Yeah, okay. It's a brave new world out there. One thing that seems to be changing now, not too long ago, I would have said I could have cited a handful of papers that sort of showed this, that typically 3 to 6 rounds of self-critique and sort of auto self-improvement seem to be where GPT-4 would kind of max out. And if anything beyond that, usually I felt like I would see that performance would decline after if you kept running it longer than that. You guys are talking about running these things for days. Maybe you could tell us a little bit of what are the budgets for this? How many tokens are we talking about? And if we translate that to retail price, what would the inference bill be for finding the the mechanism for micro drug resistance? And is there any limit at this point to how many rounds of this you can run? Or is it, you know, are we already at the point where you can just like run the thing for longer and longer, sort of potentially indefinitely?

Yeah, I think this is, it's a fascinating question. It's something that like has also like kind of like, I don't, I wouldn't say bothered me, but it's like intrigued me as well because I remember reading one of these Andrej Karpathy's post where he talks about leaving a CNN training over the winter break and then it magically led to state-of-the-art performance on some benchmark. I forget which one that was, but all that he had to do was let it run for 40 days, which is a lot of compute back then, but it was unprecedented in some ways. So yeah, I think for us with the co-scientist, I think the key thing is the fact that the system is not closed-looped. The fact that it has access to these different kinds of tools. Means that in every round of self-critique or iteration, it can bring in new information to the system that increases the entropy. And when that happens, that prevents the possibility of mode collapse and degenerate solutions from happening. So I think that is the key thing. So the fact that the system can go and browse web search, browse interesting parts of the web, take information out of it, and integrate that the knowledge that it has and do that in an effective manner, I think that is the part that leads to more computation being spent efficiently and effectively over here and helping. And it's not just websites, right? I think increasingly we'll be able to get feedback from other kinds of knowledge bases, specialized tools, AlphaFold. And so more as the quality or the surface area of the hypotheses increase, the more different kinds of feedback we'll be able to plug that into the system and I expect that it wouldn't then mode collapse and it's likely that there's going to be even more increasing value to spending more time on computation in this setup over here. But if you were to strip away that, then I think it comes down to the quality of published information in any given domain and also down to the complexity of the problem. Again, I don't know how to put a precise definition to it, but maybe you know. But if there's a problem where there's like a clear unknown that is like impossible to solve, then I think it's very likely that no amount of computation, reasoning, test-time compute is going to be able to like get you that information if the system does not have the capacity to get that information. And on the other hand, there can be like problems which are like very trivial that it doesn't matter. Like if you— you'll probably get it in the first or like a few tries and it doesn't matter. So I think there's that sweet spot over here where that problem is like within limits. but it's also not trivially easy, where spending this computation helps. And my hypothesis is I feel like a good chunk of problems that we as humanity care about today actually fall in that sweet spot where I think we can spend a lot of computation in silico and get very useful, interesting ideas and answers, which is, I think, exciting. I don't know, what do you think?

Yeah, I mean, just a small thing to add is I think also our system allows for humans to input ideas. There's other avenues to add to this entropy. I think one other important thing to think about is we're comparing pairwise every combination of hypotheses. There is a lot of variability when you compare it to maybe other papers that are talking about self-critique in a more like, we're just going to keep trying to improve the same idea. There's many directions where we're getting this variance.

Yeah, that's quite insightful. So the, and I don't know if, do you actually know the like total number of tokens for the Microbe project? I'm gonna guess it was like 10 billion or something.

No, no, I don't think it's that bad. It's probably, it should fit in within the context limits of these models. I haven't done an exact analysis, but I would think it's less than 10 million.

10 million total inference tokens for the whole thing?

Yeah, I think so.

For the whole tournament?

Oh, for the whole— no, no, no.

Yeah, I mean, like, from the time you give it the question to the time that it spits out your answer, like, what would my API cost?

It might be a little bit more than that, but we did some back-of-the-math calculations over here, and based on, like, current prices on GCP, we expect that most queries would be just a few dollars, less than $10. most queries, including all the tournament and everything inferences. So it should be fairly feasible. And it's probably just going to come down more and more in the next 6, 12, 18 months.

Yeah, that's really, especially if the performance per token is also continuing to go up, there's a lot of tailwinds.

And maybe the other thing I would say is that this is probably the most dumb and inefficient version of the system that we have. And I think we can, there's like so many things over here which we could like improve in efficiency. And also from like an intelligence perspective, we can make like much better. And so I think the bang that you would get on each token generated and dollar spent is going to be much, much more as we keep on improving the efficiency and capabilities of the system.

Are there other narrow specialist models that it has? And how important is that now? And how important do you think that will be in the future as like a source of new entropy? Because that seems like a potentially dramatic unlock, right? I mean, the model itself was already trained on most of the literature to search the literature again at runtime, helps with grounding, helps with things that maybe wasn't in the training data that it can find after the cutoff date, what have you. But the ability to actually go do simulations and bring that kind of information back that literally maybe nobody's ever run that simulation before at all, that seems like a potentially pretty big step change.

Yeah, I totally agree. And that's why I feel like this is probably like day 0 or day 1 of this journey in many ways, because we've primarily scratched the surface of information that is just written down in papers and peer-reviewed and published. But like a lot of scientific information is just not in that format. So for example, we don't publish negative results, which because of just the incentives around scientific publication. So that's like kind of like some dark matter that's hidden away. And we'll have to figure out a good incentive mechanism to have that kind of data also flow into the system. But I think that's going to be important as well going ahead. But yeah, I think more excitingly is the fact that like a lot of these papers and publications will have this supplementary data file, for example, which contains these like giant datasets of experimental data. And they contain a lot of like useful nuggets of information, which again, like You can imagine a system like this paired up with another system that came up, like a data science agent. I don't know if you saw. It could automatically go and analyze the data. And then this can generate hypotheses. The data science agent can go analyze the data and get the right kind of feedback back into the system to improve. So I think certainly at small-scale datasets, that can happen automatically. But I'm even more excited about what we might be able to do when we pair up the co-scientist elements of the data science agent and go after, say, So, ARK Institute recently came up with this virtual cell atlas. It's like 300 million gene perturbations for like 300 million cells. That's such a vast space. Even for humans, teams of humans, it would take years to explore and look at the data and see the richness in there and come up with interesting insights. But we could set up these agents to generate hypotheses, look at the data, analyze, come back with feedback. And so, I'm just super excited about the insights that it would unlock into basic biology. target discovery and things like that when we pair up these systems and set them to go on these giant datasets that we are generating right now, which even teams of humans, it's very impractical to go after and analyze right now.

Do you think— okay, here's a very high concept question that you may think is totally misguided or you might think it's the future. Obviously, over the last 3 years, we've seen a dramatic convergence of a couple of core modalities, language, vision, speech, right? For me, one of the early eureka moments where I was like, I think I'm going to study this subject for the rest of my life, or at least until the singularity, was when I realised that it's basically the same architecture doing all these things. And at the time, I was trying to make a video generation product work, and I had all these different specialist models, but I could see pretty clearly that If these fundamentally similar architectures can do these different tasks independently, then there's going to be some integration that's going to happen and the single model will be able to do them all. Now, and this has come with a very sort of viral moment, both from Flash and from GPT-4o in the last 2 weeks. Now it seems like that might happen again with the the reasoning models and the narrow specialist models. Like, right now you have a model calling AlphaFold, getting results, but there's not— this would be akin to an earlier language model, like calling DALL-E to generate an image. And there's this sort of very lossy language bottleneck that happens there that was definitely a point of major frustration for people trying to generate images that looked like they wanted them to look. And now that this integration has happened, it's like that problem is basically no more. So the question is, do you think that will happen for these other modalities as well? My sense is that an early superintelligence might be reasoning models akin to what we have today that actually have these other modalities integrated in a more deep way. So that they are not bottlenecked through API calls, but are actually able to start to do some of this reasoning in biological space or in material science space or in transcriptome space or whatever. There's a lot of spaces out there, obviously. And it seems like we already see pretty superhuman performance by the narrow models. It's like, I'm not aware of any human that can look at an amino acid sequence and intuit what the shape is going to be. There may be a couple savants out there, but certainly it's not common. Do you think that happens or what's your reaction to that possibility?

Yeah, I'm not sure. I'm not sure. I mean, to be honest, I think the API call thing is clearly, I think we're very close to being able to do that pretty well. I think a deeper integration, yeah, I don't know exactly how easy that is for For all of these kind of specialist-type areas?

Yeah, I think it actually depends on some of the incentives of people who are developing these frontier labs models. I think it makes a lot of sense to combine speech and vision and language, but if you still look at the data that's going into these models, it's primarily non-specialized public data. And then when you're thinking about like biology modalities and datasets, they are not as close to like the kind of datasets that are going in over here from the public internet over here. They're very, very different. And from like some personal experience, what we have seen is when you try to introduce, say, some of these more special modalities, even if it's medical imaging modalities, right? I mean, it's like images and natural images, and then you have medical images. They lead to regressions in benchmark performance. And so then the question becomes, okay, are you willing to sacrifice some regressions on, say, LM-SYS or some other benchmark that is generally considered an important benchmark to have your system have a little bit more capability in medical images? But it's going to obviously not be as big a fraction of your users. So I think it's a question of incentives right now. And I can see this tension in many of the frontier Small companies where when you introduce these interesting new modalities of information, it leads to sacrifice in other areas. And so you're sometimes sacrificing benchmark performance. I would argue that's a good thing. It doesn't matter so much, these benchmark performance. I think you should aim for practical utility. But in the absence of clear measures of utility, it becomes difficult to convince. And so I think it's not a question of, can we do it? I think the architectures exist. The compute exists at most of these places, but do the incentives exist? That part I don't know. And I don't know when that'll happen because it's unclear. I think today I cannot articulate, like, at a high level, I think we all agree that if we were to encode the biomedical universe, that model should be able to do a lot of interesting things. But that is sometimes conflicting with benchmark performance on, I don't know, like LMSAs or whatever else that you want to use right now. And then so it just becomes a question of incentives.

Yeah.

Okay. That reminds me of— I've brought this up a couple of times, so I'll keep it brief. But I heard Yi Tei from— he was at Reca at the time on the Late in Space podcast— talk about how the separation between vision and language models was sort of a reflection of the research history. And you at one point would have a language team and a vision team, and then it was like, maybe we can bridge these together. But then you would have late fusion models because Those things would already be sort of done and baked. And now can we get them to sort of talk to each other via cross-attention or whatever? And then it sort of became like, well, if this works, it'll probably work even better if we do it all just kind of interwoven datasets from the beginning. And I can see how that same thing might be about to play out again. I hadn't really heard so much of the benchmark thing. You're saying you've observed that in, for example, adding image capability, standard benchmarks do decline?

Yeah, I can't give you full details, but when we've tried adding medical images or, say, genomic information and try to— again, it depends on how you train these models, but we've tried to do things in a pretty standardized but maybe also the easiest way. And we've seen, okay, while obviously on the the benchmarks that are reflective of the new modalities that you're adding, over there the performance goes up. You're sacrificing performance on the main original benchmarks that are language understanding focused or vision focused. And so, yeah, you have to do this prior to optimization. And again, that also requires compute in its own ways.

But that's all in a late fusion paradigm? Like you're starting with a model that's already scoring on benchmarks and then trying to kind of—

Yeah, I mean, the easiest thing to do is do continued pre-training or SFT. So it's— yeah, I mean, like the experiments that I'm talking about are not like late fusion, but it's like more continued pre-training and SFT over here. But obviously the better thing to do would be to put all that data back into the pre-training and train all of them together. But to motivate that kind of undertaking would require you to show some benefits. And so typically how this works is you take a pre-trained checkpoint, do continued pre-training or SFT, and then if you show that, okay, your new data is helping improve performance on benchmarks that everyone else cares about, then that data goes back in. But it's a— yeah, and so that is where I think there's these tensions because these are super esoteric modalities. And so, I mean, we could spend all the time doing that Pareto optimization, throwing compute and trying to figure out the best combination, or we take this other approach where we say, okay, it doesn't matter. We'll train our own models, we'll train our own agents, and then we'll system up together. And so I think that's where we are at right now. It's a little bit of a local optimum, but I think it's still fine because it still allows us to do a lot of these things. lot of these interesting things.

Yeah, the proof is in the pudding. That's really— I'm going to remember to come back and ask a follow-up on that next time you're here. So let's see, not too much time left, and there's so much in these papers that we could cover. One thing for kind of practical utility, maybe a couple of things for practical utility for just people that are building their own systems out there. One is that this tournament-style head-to-head evaluation I've seen seems to be kind of becoming like an industry standard. I don't know if you'd go quite that far, but there's a strong trend, it seems, toward trying to surface the best ideas by doing pairwise comparisons and having some sort of World Cup-style round robin approach to doing that. I wonder if you'd add anything else to that. And then I also wanted to talk about structured reasoning as opposed to chain of thought, because I think that is one that a lot of people listening could probably go apply to their projects and get a boost tomorrow. So, yeah, maybe unpack those 2 things.

Yeah, I think the tournament one is interesting because we were actually motivated by AlphaStar, where you had these tournaments of, like, agents competing and that led to a lot of strong results in that setting. I don't know if it's an industry standard because I also feel like it's somewhat inefficient and it hints at the limitations of these models in some ways where they're not maybe able to independently score and verify ideas, but rather have to do these n² pairwise comparisons. And so we do have to do some optimizations where we have to cluster them up together, group them up together so as to reduce the computation and not do 1,000 cross 1,000 idea comparisons because that would be too expensive. So I would think that would become a little bit computationally efficient going ahead. But I think that overall ranking of things, the more you can do that in latent space, I think it's going to lead to more interesting results, better reasoning, and so on and so forth. So yeah, I expect that idea to stay but to not happen as explicitly as it's happening right now, but more of it happening in the latent space of the reasoning of these models. I'm not sure if you want to add that.

And I mean, I think regarding like the structured reasoning approach, like I think there's like practical reasons for it, right? Like when we have, you know, 2 agents that we want to talk to each other, like it helps to have like a data structure that we're passing, right? And that just makes like the engineering of it itself like a little bit easier. But I think beyond that, comparing to like just like asking the model to like, you know, in a chain of thought style, like just reason about this versus defining a certain reasoning structure. I think the advantage there is we can actually kind of better enforce it to follow a certain path. So in our case, we wanted it to do this long analysis before kind of going into higher-level management goals, before then finally forming its management plan. And being able to define that structure forced the model to take that path through its reasoning rather than freeform, if we allow it to do it freeform, maybe it starts to form its management plan before it has done these kind of higher level steps that we want it to go through.

Yes, it's basically to just try to describe this for people that might want to implement it. I mean, for one thing, it takes advantage of another notable feature that models have gained over the last year or so, which is that we can now specify as part of an API call, this is the exact JSON data structure that you are supposed to return. So that's huge because it makes it really easy to set that up and then get something back. It kind of is a little bit like airline checklisting, sort of where you're just like, okay, I want you to absolutely every time go through these steps. And if you do that, we're confident you're going to get the better results in the end versus just kind of walking out there and randomly walking around the plane and coming back and saying, yeah, it all looks good to me. Right. So the intuition for that is pretty simple. How dynamic did you make that? Because I've never actually done a dynamic structured output.

Yeah, I think in our case, I think we tried a lot of different structures and strategies for you know, generating the ultimate management plan. I think there were, like, when we tried to get a little too, like, fine-grained with, like, you know, okay, first, like, summarize the patient's, like, you know, chief complaint, then summarize, like, when we tried to get, like, a little too granular with that, I think it's— I think, I think having the flexibility of, like, just these, like, kind of higher-level things, like analysis management goals, like things that are Pretty general tended to lead to better management plans. Of course, this is all under our own auto-evaluation and vibe checks, as I mentioned. So, you know, that's maybe up for debate. But I think, yeah, and I don't think it was like super— so it was dynamic in that sense. Like, you know, it can have any number of analysis items. It's like a, you know, a list of however many items, any number of management goals. But I think we try not to constrain it too much, just constrain to the point of Like, we want it to go through a certain reasoning structure. Yeah.

Okay, so structured outputs, people, don't overdo it, but definitely use it. I mean, that was one of the things that seemed like it really drove a pretty big lift. Going back just to the comparisons for one more second, Vivek, are you basically saying that you think in the future it won't have to be so head-to-head and instead it'll just be like, here's 10 things, pick the best?

Yeah, I would hope that the— so in some ways what we're trying to do is like force the model to explicitly do the tree search, right? You know, come up with the new ideas, like go to different nodes, and then do the comparison over here. But maybe what I'm— so it's almost like, okay, like you come up with ideas, you write them down, then you review them, and then you figure out what's the best that's happening. But like the question is, okay, can all of that happen in your head in some ways? Like does it have to be explicitly written down? Yeah. You have to explicitly generate all those tokens. Can you do something in the way you set up the architecture itself or in some other mechanism where all of that happens in the latent space itself so that you're more efficient with the tokens that you're generating? So I feel there's an inefficiency right now. I mean, sure, it helps with interpretability and other different aspects over here, but I think there's a lot to be gained by encoding that tree search within the latent reasoning of the models, and we don't do that as well right now.

Yeah. Okay, cool. That's helpful. Maybe the last 2 things. What do you think would happen if you just prepended a question identification agent to the co-scientist and just had the thing kind of run in a loop where the first thing it did, instead of taking a question from a human scientist is just like, go out on the internet and search around and come up with an interesting question for itself and then just try to answer its own question. Is there anything about that that you think wouldn't be effective?

I feel like in some ways we like to talk about the concept of root node problems at Google and DeepMind, and we felt like Once you have a question, like generating novel original solutions to that is a root node problem. But in some ways what you are describing over here is an even upstream root node problem. Like, how do you ask the right question? And I feel like the day we get AI systems to reach that, then I think that is the day we can truly say like, okay, we have geniuses in data centers. I think that is like going to be the most impactful and important unlock. And I mean, my feeling is like. There already should be like decent capability in these models to like, you know, go surf the internet, read information, and figure out what are the right questions to ask. Um, so yeah, I mean, if it's okay, we'll go and give it a try and get back to you.

Um, have you had a chance— this is a, a bonus— how have you had a chance yet to try this with Gemini 2.5? It seems like from my qualitative assessment, it would be a lot better.

Yeah, no, I think that's the exciting part because whatever we've described in the paper was all Gemini 2.0. So it should be coming up very soon, but yeah, we're super excited about that.

Feels like the path to geniuses in a data center is honestly pretty clear at this point, which is a crazy thing to say, but you know, do you see, are there any like, you know, big barrier questions that you feel are You know, just fundamentally unanswered still? You know, programmers often call it a simple matter of programming. Like, it's going to be work, but, you know, we can make it work. Is that kind of the mindset right now for you guys, or are there questions where you're like, we really don't have a good answer to how we're going to get over that part?

Like, we have all the building blocks over here, and so we'll probably build something that will look very close to what you're talking about over here. Whether that's the most beautiful one, the most elegant one, we don't know. But does that even matter? I think it doesn't. So, I think that is why it's truly exciting, where I think we have line of sight to one solution, which feels like will get us where we want to. And that in turn is going to lead to a lot of new unlocks. And so, yeah, I would say, I think for the next couple of years, at least, it to me feels mostly an engineering challenge rather than trying to answer some fundamental unknowns.

Yeah. And of course, the other big challenge is going to be the social challenge of Introducing this stuff to the world and getting scientists to pay attention. Maybe to close, you want to talk a little bit about what you're doing in that regard? Because I was excited to see that we're now getting to the point where you're inviting scientists to reach out and partner with you on this. And also going into— I don't know if you would officially call it a clinical trial, but like something in the actual field of medical practice with real patients too. So tell us what you're doing on the deployment side.

Yeah, I think that CoScientist is a little bit easier for us to deploy. I think there are maybe less questions around regulatory and things like that. And so it's— so I mean, the only thing that maybe bothers us a little bit is that the system is highly capable. And so there are also many ways in which it can possibly not do so well.

Right.

And so we just want to ensure that as we scale up the system, we do that in a responsible manner. And so that's why, like, you know, we have this Trusted Tester program. We've already been working with like close to 100 scientists right now across the world. And these are all like world-leading experts. And with the Trusted Tester, we want to invite like more organizations. And so our hope is that like we can do this in like batches and in waves. And then every batch we get like feedback, we identify like the weak points of the system, we improve it and make it better for the next batch of scientists. And so Yeah, this shouldn't take too long. I think by the end of the year, I'm hoping that, yeah, if I'm optimistic, millions of scientists around the world will have access to this tool. And hopefully it raises the bar, the ceiling for all of them and helps them do more creative and interesting work. I think the one with Amy is a little bit more tricky. It's obviously a more complex space. But again, there's a path, there's a known path to taking such systems out there in the real world that can give diagnosis and treatment recommendations. And so We are very excited about the clinical trial that we have coming up. So it's going to be, I think, one of the first studies of its kind where an LLM-based system is going to be interacting with real patients. And the nice thing about the setup that we have right now is we are deploying it in the clinic where there is sufficient presence of clinical experts who can oversee the system and provide oversight. And so that is a very safe environment for us to deploy the system where there's sufficient number of doctors who can take over if something goes wrong. And our hope is in that study, like not a lot, lot like goes wrong. So that allows us to dial down the amount of like oversight that's needed. And so yeah, I think if things go well, then we'll probably scale it out to more centers, like reduce the amount of expertise and also introduce like more net new capabilities into the system that we are going to trial and like make them more patient-facing. So yeah, I think that's, that's the exciting part where I feel like, like the research has progressed quite a bit. And so it's now time to like see, like take it for a drive in the real world.

It's exciting times, guys. Really outstanding work. People should be paying more attention than they are, and hopefully we'll put a little dent in the consciousness by bringing some more attention to this. But just really mind-blowing, really mind-blowing stuff, and quite a series of work that you guys have put out. Anything else you want to share in parting, or any other thoughts you want to leave people with?

No, I mean, it's been a pleasure to work on these projects, obviously, and we have a lot cooking still. So, you know, I'm excited. I'm excited for the future of this. Yeah.

Yeah. And likewise, I think for me, like talking to Nathan, I don't know, it's like just, just a lot of fun. And yeah, I mean, it's a real pleasure, like 4th time over here. I think there's gonna be at least like one more where it's big enough that we'll come back again and talk in the next few months. But yeah.

I just hope you don't get too big for me. That's my, that's my hope.

Nah, I'd love to be here. I think it's so much fun.

Cool. Well, really appreciate it. Again, fantastic work. Vivek Natarajan and Anil Palefu, thank you for being part of the Cognitive Revolution.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.