Bioinfohazards: Jassi Pannu on Controlling Dangerous Data from which AI Models Learn

Hello, and welcome back to The Cognitive Revolution. Today, my guest is Josie Penu, assistant professor at Johns Hopkins, who recently co-authored an important paper calling for the creation of access control systems meant to prevent the dissemination and misuse of functional biological data from which AI models could learn extremely dangerous capabilities, such as the modification or even de novo design of highly contagious, and deadly viruses. We begin with an overview of the biosecurity landscape today, including how new viruses are detected, how patient data is aggregated and analyzed in the context of a new threat, and what the pipeline from DNA sequence to vaccine candidate looks like today. The good news is that we are able to design new vaccines amazingly quickly, at least for viruses that are similar to others we've seen. But there is, unfortunately, a lot of bad news as well. In 2012, for example, 2 research groups independently published results showing that wild-type bird flu, which already had an estimated 60% fatality rate but couldn't spread between humans, could become mammal-to-mammal transmissible with just 5 mutations. Such gain-of-function research has been broadly defunded since the COVID pandemic, but it does remain legal. And visibility into the experiments that private labs are conducting is low. Governments, Jassy says, aren't likely to develop bioweapons capable of causing pandemics for the simple reason that, short of vaccinating their populations in advance of an attack, they can't realistically expect to control them. But with AI capabilities crossing critical thresholds month by month, the threat from extremist groups and even lone actors is quickly moving from a theoretical to a deadly practical concern. Consider that Geoffrey Irving, chief scientist at the UK AI Security Institute, recently highlighted for me that today's frontier models can troubleshoot laboratory experiments from a cell phone picture better on average than PhDs. And in just the 10 days or so since we recorded this conversation, we've seen Andrej Karpathy's AutoResearch framework demonstrate that AI agents can run and make research progress for days on end. Even more to the point, Anthropic just reported that Opus 4.6, when faced with a benchmark challenge that it couldn't solve, spontaneously located the full benchmark dataset on Hugging Face and then figured out how to decrypt the solutions, which were encrypted in the first place for the purpose of preventing the answers from leaking into training data. And it did this all in order to get a single question right. With reasoning AIs already capable of spontaneously overcoming such barriers to information, I think we should expect that future research agents will find and exploit any signal-rich data that exists anywhere on the internet. And with the smallpox sequence and the horsepox synthesis protocol already published online, and biological data poised to grow super exponentially in the coming years, we have real reason to worry and ample cause to get serious about implementing data controls before the situation gets truly out of hand. Again, though, there is good news. Recent work by the teams behind the EVO and ESM families of biofoundation models showed that strategically excluding key datasets such as the DNA sequences of viruses that infect humans, dramatically reduced models' performance on dangerous tasks while leaving their desirable capabilities intact. This means that the vast majority of biological data can remain open source and open access. And indeed, Joshi and co-authors' proposal for a biosecurity data-level framework that echoes the existing Level 0 to Level 4 biosafety framework for physical labs would subject only an estimated 1% of data which connects pathogen sequences to dangerous properties, to any additional restrictions. And even then, structures such as trusted research environments, which allow researchers to run code on data without transmitting that data from its secure location, would still support valuable research. Once again, despite my personal history as a lifelong techno-optimist libertarian who broadly believes that data wants to and ought to be free, I find myself eager to support these control measures. Of course, that's not the only opportunity we have to improve biosecurity, and toward the end we also discussed the broader defense-in-depth strategy that biosecurity experts recommend: delay, deter, detect, and defend, which includes mandatory pre-synthesis screening of sequences by DNA manufacturers, investment in wastewater monitoring and other passive global pathogen surveillance, and practical frontline defenses like PPE stockpiling and far UV sterilization. All of this is in everyone's shared interest, but it does require leaders to see beyond the current news cycle for long enough to make it happen. I certainly hope they do, but I also recommend taking individual action where you can, both to improve your own personal safety and to support the consumer market for biosecurity products. My wife and I, for example, at our friend Jeff Kaufman's recommendation, recently purchased an AeroLamp far UV light for use in my son's hospital rooms throughout his cancer treatment. And I'd welcome additional suggestions for other products that could help us minimize disease burden today while also serving as a sort of private insurance against pandemics, if anyone has any recommendations. For now, I would simply emphasize that by default, we are fast approaching a world in which a rapidly growing number of people, and perhaps autonomous AIs as well, will have the ability to create deadly, transmissible, self-replicating viruses, which could dramatically alter the trajectory of human history. And it really does seem like we should do something about it. With that, I hope you are properly alarmed by this scary but solutions-oriented conversation about the sorry state of biosecurity and the rapidly rising threat from bio-savvy AI systems with Johns Hopkins professor, Josie Penu.

Josie Panu, assistant professor at Johns Hopkins. Welcome to The Cognitive Revolution.

Thanks for having me.

I'm excited for this conversation. I think I'm going to learn a lot from it. So let's dive in. Recently, you co-authored with a number of others a call for controls on biological data. We've heard, I think, quite a bit about the possibility that AI systems of various kinds could create new sorts of biorisks. And this is one attempt to put some controls in place to hopefully cut that off at the pass before it really becomes a big issue downstream. I want to get into that from a bunch of different angles, but I think actually it would be helpful to maybe take a little bit of a step back and lay some foundations, because I think people who follow this feed, they know a lot about the AI side. They probably on average don't know nearly as much about the current state of play when it comes to biological data writ large. And so I thought it would be helpful just— and I'm very curious actually about some of these things I realized I don't know as I was preparing for this. So for starters, if you could take us into the moment which we had not so long ago and hopefully won't have again, but we very well might, where all of a sudden there is a new outbreak of something, right? Something that we've never seen before. We don't know what it is, and people are concerned, and patients in a particular hospital or a particular city are showing up with concerning symptoms.

What happens?

How do we turn that sort of initial small patient population into actual knowledge about what it is we're dealing with?

Yeah, yeah, I think that's a great place to start, and I think just to take a step back, it's important to realize how does society come to the conclusion that there is a new virus circulating? What is our mechanism for making that decision? Because we don't have a global or national alert system for this kind of thing in the same way that you have a radar system for ICBMs. Right now, we largely rely on symptoms. So patients get sick, they visit a hospital, doctors get concerned, they run a series of tests, and doctors always start with tests that are common. So things like influenza, RSV, rhinovirus, and we're just collecting samples by swabbing their nose and sending it to the hospital lab. And it's when these tests are negative and that if the patient is very sick, then we'll go ahead and run further tests. And we learned that— we learned during COVID-19 that this whole process is pretty lengthy. It works well for familiar viruses, but it does not quickly detect new viruses. And in the case— I'll just mention the case of influenza because that's where the situation is a little bit unique. We've had several influenza epidemics in the past, so we do actually have a global system to collect new influenza sequences, but that also operates on a contribution model. So the national lab has to send up that sequence to the global repository, and it's an active submission process. We don't have a passive global alert system, and we can talk about the benefits of having something like that later. But once the community has decided, okay, it looks like there's a cluster of patients where there's a new virus, we need to figure out what this virus is. That's when we start to do these more exotic tests like metagenomic sequencing to try and figure out what is the sequence of that pathogen. And that's what happened during the COVID-19 pandemic. So we think based on research that COVID likely emerged around late November to December of 2019, but then it was fully sequenced in January. And the decision to share that sequence was— the way it happened was that a researcher sequenced the virus. This was a researcher in China. They publicly shared sequence. They didn't request permission from the Chinese government, which is pretty different from how it might have happened for influenza virus. And that's when we started— kicked off the process of designing diagnostics and trying to scale those.

I recently had an experience of My son has had cancer. People know about this if they've listened to this feed. I went through this process of sharing data with the broader medical establishment. So somebody, I think it's like a big part of their job based on how much time they spent with me, came by and said, hey, I'm from the sort of data sharing world, and I'm here to answer all your questions and get 1,000 signatures on 1,000 pages so we can hopefully use this data for the betterment of all humanity. And I said yes to all that stuff. And then I've still been getting stuff in the mail asking me to share data again with particular studies or groups or whatever. And I'm wondering if when it's a pathogen, if that's maybe different, because I guess what I'm sharing in the case of my son's case is information very specific to him, whereas you could think about it differently if it's a pathogen. It's not your pathogen, right? It's the world's pathogen in some sense. But how much consent or opting in do patients have to do, if any, to get this data from them and into the higher levels of analysis?

Yeah, what you're pointing to is the fact that we have very different systems for dealing with individual-level risk versus societal-level risk. So we have a really strong set of protections around making sure that your son's data isn't inadvertently shared, that you've consented to everything, and we're thinking about the privacy risks of all of that data. And that's really to protect your son, the individual patient. But then when it comes to societal-level risks about a pathogen where basically the whole globe is your patient, we don't have as good mechanisms for figuring out how to protect society from those risks. And we don't have as good mechanisms for protecting the data that could lead to those consequential outbreaks. But yeah, just to focus on the clinical data, that very much depends on where you are, what country you're in, what system you're operating in. And in the US, as it sounds like you've experienced, our system is very fragmented. It's different within different states, within different networks, whether it's a public community hospital or a private hospital. And you— those different— the fragmentation results in different people having to ask you for permission over and over again, because there isn't a single unified repository for all of that data. It is a bit different in other countries. So if you look at the UK, they have the National Health Service, and what the UK has been able to do is create a research platform called OpenSAFELY. And this was something that was stood up during 2020. It provides researchers access to clinical data for 95% of the UK population. And it's really interesting because it doesn't require giving the data to the researchers, it forces the researchers to come to the data. So the code comes to the dataset and you can submit your code to the platform. You never have to see the private data. It's secure the entire time. And there's been a lot of positive reception to the platform of OpenSAFELY, but we don't have an equivalent in the US where everything's unified in that way.

Yeah, that's interesting. We'll come back to maybe the different levels of protection or restriction that certain different datasets should have in your mind. But let me just stay on the response narrative for a minute. So I know that it sounds like it was actually maybe longer between when the virus first emerged and when it was first sequenced. than it was from when it was first sequenced to when the vaccine was initially designed. I've read a couple of these magazine pieces that tell the story of, I don't know, 48 to 72-hour period where certain researchers got the sequence, they were able to do some analysis on it, identify a particular protein, which I think was the spike protein, and then plug that into an mRNA platform and My understanding was that the final vaccine that I got wasn't that different from the initial design that they put together in a couple, probably pretty long days. I think it was even in maybe as early as February of 2020. Has that pipeline changed at all? Obviously, there's been so many different ML models that have come out over the last 5 years that predict shapes of things better and what's going to bind to what better. But I don't know, that's pretty hard to beat in terms of timeline. Would you say if that were to happen again today, would that process look much different from how it looked then?

Maybe first I'll talk about what happened during the COVID-19 pandemic and those different timelines. And then we'll, we can reflect on like how this will look in the future with AI. So you're completely correct that during the process of designing COVID-19 vaccines, and I'll focus on the mRNA platform vaccine as a primary example. The design process, the computational steps where you were looking at the spike protein, working backwards to figure out what is the mRNA sequence that you're going to use for your vaccine design, that process was very fast. But that was clearly not the bottleneck. There were many other steps that took much longer. And so, yes, the process of knowing there was an outbreak and then sequencing the pathogen, figuring out what pathogen we were dealing with, And then, then yes, those steps took quite a bit of time. But then there was also a whole body of research that researchers relied on that I think doesn't get enough airtime. Because once researchers realized that we were dealing with SARS-CoV-2, they were able to look at this body of research that was done on SARS-CoV-1, which was the original variant of acute respiratory syndrome that happened many years prior. It didn't spread into a global pandemic, but there were patient cases that were isolated across the globe, and we actually managed to prevent that from becoming a global pandemic. And it was that research that resulted in researchers knowing that the spike protein was extremely important. So then they were able to say, okay, we know spike protein, it's important, let's computationally design the vaccine. That part was very fast. And then it was turning that into an actual vaccine. So all of the clinical components, regulatory, process, the different clinical trials that you have to go through, and then distributing that vaccine globally. Those steps took a lot longer than the actual computational design. I think that now with advancements in AI models, people are very optimistic about being able to design new proteins, design antibodies, design vaccines. I would say that there still is going to be the bottleneck of scaling in the physical world. Clinical trials still remain a huge barrier. scaling and deployment of vaccines across the globe was a huge barrier. And so figuring out if AI can speed up those steps will be really useful. And it's perhaps the neglected component of the overall pathway.

One thing I've learned in looking at the question of security at frontier model developers in the AI industry is that there's, you know what, in looking at like responsible scaling policies, right? There's a lot of levels to the game of security. And the different levels seem to correspond to different actors that you might be concerned about and how hard it would be to prevent them from doing bad stuff. So for the likes of DeepMind and OpenAI and Anthropic, the general consensus seems to be that if a determined nation-state actor, China, were to want to steal the model weights, they would probably be able to do it, and there's not too much that could be done to prevent that. How would you map the sort of threat landscape when it comes to the biorisk side? Is it a similar thing where we have random crazy people versus somewhat more sophisticated groups of people versus all the way up to nation states? And what do you think is like reasonable to expect we can actually stop with all of the measures that we'll talk about potentially developing?

Yeah, it's a really important question. I think with, within the realm of biosecurity, there are a lot of different threats that people refer to when they're thinking about chemical and biological weapons. There's things like toxins, small molecule toxins, protein toxins, then there's organisms that are not transmissible between humans, things like anthrax, where we have reliable countermeasures, antibiotics that work against those things. And then there's the more extreme of pandemic threats, pandemic viruses that are novel. We've never seen them before. We don't have diagnostics for them. We don't have therapeutics or vaccines for them. And when you think about that spectrum, the consequences of— the potential consequences of a pandemic virus are far higher than many of those other threats. This is all pretty obvious to us now after having lived through one. But it's also interesting that a pandemic virus is not a particularly desirable weapon for a nation-state. It's not targeted. It's not easy to necessarily protect your own population. You'd have to design a vaccine and vaccinate your entire population. It's hard to do that without someone noticing. And so I think in general, nation-states are not the primary actor that one is considering when they think about pandemic threats. It definitely is more so the folks that are not motivated by rationality. So smaller groups, terrorist groups, potentially lone actors, those are the folks that people really are concerned about. And so that's why a data control mechanism is most likely in the interests of all countries. I would say that China is equally invested in making sure there isn't a future pandemic as the United States. So I'm hopeful that there can be some international cooperation, or if at least not cooperation, acknowledgement that data controls benefit both the US and China and nation states globally. That's what I would say. Yeah. In terms of data controls and how can you prevent those kinds of actors that I outlined, lone actors, smaller groups from getting access to your datasets. I think that controls are meaningful there. I think that currently the default is sharing that data publicly available for anonymous access. It's extremely easy to access, and even putting minor barriers would make a difference. And the other important thing to consider is we want to make sure that defenders, or people who are advancing countermeasures research, advancing virology research, they have access to that data, and you're limiting access to people malicious actors. And so controls can kind of do that differential privileging where you're privileging defensive use cases, you're limiting the offensive use cases. You can track who's using it, give access to that crowd and limit access to others.

Hey, we'll continue our interview in a moment after a word from our sponsors. So can we maybe just map out the data landscape as it exists today? I kind of want to do this for both the data landscape and the models that are obviously spawned from the data. On the data side, I've heard it said many times that you can find the smallpox sequence on the internet. And then there's the question of, if that's true, why hasn't that turned into a crisis already? And I have heard various accounts. I'd be interested in yours. But then more broadly, that's like a known sequence for a known problem, probably You tell me, but it strikes me that it's a small enough amount of data that's probably pretty hard to control or clean up from all the places that it might already have been replicated. It's a little bit hard for me to imagine a world where that's been scrubbed so thoroughly that somebody who wanted to go find it wouldn't be able to find it. But maybe you'd have a plan for how we could get there. But then if we expand out the scope of data of concern, There's tons of biodata in general, right? And I know most of that you're not looking to restrict. So how would you draw the sort of, I don't know if they're concentric circles or not, from the most narrow, this is the smallpox sequence, which we probably shouldn't be passing out too freely, to somewhat larger. And then beyond that, everything that would be fine. How would you characterize those classes of data?

Yeah, let's start with just the general broad categories of data. The most abundant biological data that currently exists is sequence data. We are currently swimming in petabytes and petabytes of sequence data, a lot of which we don't know what it does or the functions of those different sequences. And it's just because it's been extremely easy to collect and now sequence that kind of data. There's something called Carlson's curve, which is like the equivalent of of Moore's Law for biology and DNA sequencing. And it's just actually shattered Moore's Law because it's become exponentially cheaper to do DNA sequencing. And so that's resulted in a lot of passive DNA sequencing collection and just sequencing everything. And that kind of data is available in government-supported repositories. So things like GenBank, which is supported by the NCBI, part of the NIH in the US, And that alone has 40 petabytes plus of just unannotated, raw, poor quality, frankly, DNA sequencing data. And, yeah, that is a large part in why there are efforts to build AI models using that data, because it's abundant. When we think about other types of data, there's protein sequence databases, and then there's the protein— PDB, Protein Data Bank, which is what formed the basis of AlphaFold, and that contains protein structure data. That data was collected by hand over the years. I'm sure a lot of us have heard the story of painstaking efforts to do experiments to figure out protein structure, where one grad student would— that would be their whole PhD project. And that dataset is actually very small. It would definitely fit on a thumb drive. It's like less than a terabyte of data. And so those, like the different types of data we have access to for biology, they're disparate, they're different types, and they're different sizes. So when we're thinking about what, in that whole landscape, what kinds of data might be of particular concern, I would say that it's currently an open question. There are some that think that you could train an AI model on genetic sequence data alone and get a pretty functional model. And I think that models like Evo 2 that have done this, where they've just trained on DNA sequence data, they can perform well on protein-related tasks. They can span scales, they can do genome generation. So there's optimism that you could do quite a bit with just sequence data. But I think that there is this, this other view in the community that really genetic data is observational. What you really need to advance biology is some kind of data that gives you insight into causality. And that's where things like perturbation data or knockout datasets, where you're systematically knocking out different genes of a virus and then looking at how that impacts the function, or you're systematically looking at how viral proteins bind to human proteins. That's— I'll just broadly call that functional data that gives you some insight into causality. And there's thinking that incorporating that kind of data into training AI models is what's really needed to get you over to making functional biological constructs that are viable in the real world. And so what we're proposing in terms of our data controls are that, you rightly said, the vast majority of data should not be under control. I think that there's actually been a huge effort in biology to make data open access. and more openly shared because that advances research overall. So fully supportive of that. And the controls that we're proposing are really on functional data that gives you insight into important features of viruses that frankly the U.S. government has recognized result in information relevant to whether this virus is pandemic capable or not. So transmissibility, Virulence, or how deadly it is, things like immune evasion. So can you modify a virus so that it gets around an existing vaccine or gets around your own immune system? Those are the kind of features that the US government actually already tracks for wet lab research. If you're proposing a wet lab experiment that intends to enhance a pandemic virus to make it more transmissible, more virulent, or to evade your vaccines, that's something that the US government wants to know about and is going to ask you whether or not it's a good idea. So doing it in the computational domain is just extending that a little bit further. We're really proposing focusing narrowly on that kind of data. And the one other thing that I'd add is that I completely agree, going out and scrubbing the internet of data that already exists is not going to be possible. It would be a Herculean effort and probably not worth the effort. And what we're proposing is controls on data that's generated in the future. new datasets. I guess my thesis is that now that we know that AI is quite promising for biology, there will be huge investment in creating new datasets. We're already kind of seeing this with the US government's Genesis mission, with the OpenAI Foundation's commitment to spend billions on datasets, the Chan Zuckerberg Biohub as well. And so as really large-scale efforts get underway to generate not just observational, but also causality-related information, perturbation datasets, that's where we want— we're suggesting if that's done on pandemic pathogens, that data should probably not be shared for completely anonymous access. You should track who has access to it and have some controls around it.

So going back to the people doing wet lab experiments on gain-of-function-style premises, I'd be interested in your take. To put my cards on the table, I think that's not a good idea. When I look at the timeline from the rise of the virus to the sequencing to the vaccine design in the COVID case, and I do take your point that there was some prior knowledge there that accelerated things, I'd also be interested to hear how much. If it weren't for that sort of COVID-1 knowledge, how much would that timeline have moved? But I guess my zoomed-out and somewhat ignorant view is that didn't take that long. Certainly the clinical trial part took a lot, and the distribution, manufacturing distribution, took a lot longer. So if the argument is we want to do these experiments now because we will be able to shorten the timeline in the future if something like this does happen, to be able to respond to it, I would say you're not really taking the bulk of the time out by quickening the pace to a vaccine design. And also, I'd hate to see you let it loose, so maybe just shouldn't do that. Do you see that the same way? And then I guess another question would be, an obvious extension would be like, should we apply the same reasoning to certain kinds of data generation in the first place? Is there a certain kind of dataset that we should just say maybe we're better off not scaling? It'd be interesting to know exactly what it is in these viral sequences that cause transmissibility or whatever, but we are creating something there that in a sense can— it's obviously a little bit more upstream, but could in theory escape in a similar way. Yeah, let's start with what's your take on gain-of-function research in the wet lab, and then does that same analysis apply to the data generation side?

Yeah, excellent. And I think you can say your view with more confidence because I think it's a very reasonable view. And there was a lot packed in there, so if I missed anything that you just asked, feel free to just flag it to me. But to just provide some background on what is gain-of-function research, perhaps— I don't know if you've talked about this before on the podcast— I'll give an example from 2012. So in 2012, there were 2 experiments that were done by 2 different research groups looking at avian influenza, which at the time was thought to have a 60% case fatality rate, so highly lethal, but was not human-to-human transmissible. So there were cases of humans getting avian influenza from animals, but it was not resulting in a global pandemic because it didn't efficiently transmit between humans, which is a happy accident for us, frankly. And what the researchers were doing was they did animal experiments in ferrets where they intentionally increased the transmissibility between ferrets of that virus. And ferrets are the known mammalian model. They're meant to represent human immunity. And so the hypothesis was that they were creating a human-to-human transmissible version of this highly lethal virus. And of course, when these experiments were submitted for publication, I believe they were simultaneously submitted to both Science and Nature at the same time. Those journals received those publications and alerted the US government, wondering frankly what they should do with these results, because the manuscripts included the specific mutations that would be required to create that level of transmissibility. And one of the groups found that it was only 5 mutations in the avian influenza virus that got you to human-to-human transmissibility. So with that work, and so that's the kind of work that people refer to as gain-of-function research. The technical term is dual-use research of concern or enhanced pandemic pathogen research. And that work at the time, it was recommended that those publications, or at least the details of the mutations, not see the light of day. And there were 2 concerns there. One is the concern that you raised, which is Wow, humans are working with these kinds of pathogens in the lab. We know that there's human error. And what happens if someone's dealing with that pathogen that they've just created to be more transmissible, they themselves get infected, leave the lab without knowing it, and trigger a global pandemic? That's a legitimate concern. There have been instances in the past where samples of really concerning viruses have been found in settings like the CDC. So past examples, the CDC found vials of smallpox in a freezer. They didn't know they were there. They were still viable. So there's supposed to be only 2 places in the world that have active samples of smallpox, and these were not known to be there. So I think overall there's concern about lab accident, human error, and that was one of the major concerns that's generally falling under the bucket of biosafety. The second concern was that aside from dealing physically with the pathogen, there was the concern about the information related to the experiments. So not only how the researchers did this and how someone else could replicate that same effort, but then too the exact mutations that would be needed and whether or not that was information that should be in the public domain. And this relates very much to what you described, which is the horsepox synthesis protocol is in the public domain. The smallpox sequence is in the public domain. So theoretically, someone could put those 2 things together and try to create smallpox, even if they weren't able to get access to the physical specimen themselves. And overall, there's just lots of information about protocols for doing reverse genetics or other ways of rescuing live infectious virus for pandemic pathogens. So in the wet lab field, this has actually been a huge debate for years and years with regards to what do we do about this information? Should it be controlled in some way? There— and so far where policy and regulation has come down is there's a focus on controlling the physical specimen, and there's a focus on preventing experimental work to increase these concerning characteristics of pathogens. But there's been— it's considered infeasible to try and go back and— scrub the internet of data that's already out there. And so we really have to try and figure out a mechanism for deciding this in advance before it's already out there and we, we can't do anything to pull it back.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database. that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires. Based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he's currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

But it's still not illegal to do this, is that right? Like you can, Do you need any special permission, or is it an ask-for-forgiveness-not-permission regime that we're on with such gain-of-function research, even in the physical realm?

Yeah, this is— it's a really good point. In the case of the 2012 experiments, there was no legal— there was no law that those researchers were breaking. The mechanism that the U.S. government at least has used in the past has been using regulation. Essentially, if you receive funding from the US government, you therefore have to follow certain policies. And so this policy around not doing research enhancing pandemic pathogens is one way that the US government has tried to do this. There are some laws on the books for dealing with controlled pathogens, where it's the Federal Select Agent Program. For example, if you want to handle anthrax samples, You have to be a registered lab that is tracked under this program. But yeah, that's slightly separate. I would say the other thing to consider is work that, for example, seeks to go into bat caves. People are collecting samples of viruses where there's a suspicion that those viruses could be pandemic capable. They sample them. in those caves. This is called virus hunting. They bring the samples back to the lab, they manipulate them, they try to characterize them. And this is also something that the US government and other governments used to spend money on. But after the COVID-19 pandemic, the fallout from that, the lab leak hypothesis, the political dynamics around that, a lot of that work has been defunded, but it's not explicitly illegal. Is—

I don't want to get too bogged down at this particular point, but Is there a good reason for that? I do understand, of course, that we benefit tremendously from biomedical research broadly. I could imagine you might say, actually, the border is a little harder to define and it can be a little fuzzier, so it's hard to legislate. But if this is one that we're sleeping on without a really good reason, it might be time to start writing our representatives. So Is there a good reason that this isn't more controlled than it currently is?

I would love to see it more controlled than it currently is. I think that the real reason that it isn't is because governments are good at legislating things that happen often. And what we're dealing with are pretty rare instances that certainly would lead to extremely high consequence harms, global pandemics, things that we don't want to see, but they just don't happen very often. And so the push for policymakers to— this is a live issue, this is something that needs to be legislated— that comes and goes very quickly. And we already saw with the COVID-19 pandemic, there was a lot of concern. There still is debate as to what was the origins of COVID-19. We haven't resolved that question. The WHO director actually just a couple of days ago put out a statement saying that we still need to do work on resolving this question. But the policymakers have moved on to more pressing issues because that's just the nature of policymaking, that they have to put out fires today. I think that in reality, what we need is we do need both national and international rules. So right now, the WHO has rules where they say only Russia's Vector Institute and the US CDC those are the only 2 places in the world that can have access to smallpox. That's a great initiative, but that did not prevent a researcher from unilaterally publishing the step-by-step protocol for how to synthesize horsepox, the close relative to smallpox. And so what that kind of highlights is as synthetic biology, virology, biomedical capabilities advance, We need some better way to make sure our regulations keep up. That's a concerning topic. But I think so far, you know, the reason that we don't— this isn't a live issue or why we're not kind of thinking about it day to day is because it still does require a lot of expertise to synthesize any of these pathogens from scratch. It really is something that you need to have a lot of background in. But this is where the concerns related to AI come up. And so in terms of like the different types of AI models and whether or not they provide uplift and what kinds of biomodels could be used to do this, this is such an evolving and kind of open question that people are trying to figure out. I think the hard part is that it's moving quite quickly. And so it's, it's hard to see how policymakers can keep up, but we're working on it.

How much like actual wet lab gain of function research do you think is going on today? Has it been dramatically curtailed by these sort of strings attached to funding and general awareness in the community that it's maybe not a good idea? Or do you think there is still a lot going on? Because I guess another reason that there might not be a law is, well, everybody quit doing it because they realised that it was a bad idea, and who needs to make a law against something nobody's doing? But is that the case, or do we have any even way of really knowing how much is going on?

So with regards to wet lab gain of function research, I guess I would first want to say That the kinds of research that we need for future vaccine design, like determining the spike protein is important, that's like how we advance our ability to create vaccines for new pathogens. That kind of work doesn't require gain-of-function research. Gain-of-function research, the kind that we're talking about, is very narrowly scoped, and it does not require enhancement of the transmission of a pathogen or making it more virulent or making it escape the immune system, those kinds of experiments are really not needed for the vast majority of the kinds of advancements we would want in biomedicine. So with that in mind, I would say over the past few years since the COVID-19 pandemic, a lot of this work has been defunded and has been reduced just by US government funding mechanisms. Our blind spot is the work that's happening in Private labs. We don't actually have any legal mechanism for going into a private lab and determining, are they doing certain kinds of pathogen research other than whether they have registered under the federal Select Agent Program. But there have been instances of laboratories in California most recently where they're handling certain types of pathogens that they really frankly shouldn't be, and they don't have the containment protocols for. Yeah, I'll pause there. Overall, I think We're in a better spot than we were. I think people have recognized the downsides of this research and the risks. And certainly governments are paying a lot more attention to it.

It's funny. It echoes in a way the reduction in bad behavior that we usually see from one generation of large language model to the next, where it's like, we recognized that this was a problem, did some stuff to try to curtail it, and we reduced it about 90%. Great news. The other 10% is out there for future work to contend with. Dizzying in the gain of function case. Okay, you mentioned, let's go with your segue. So there are different kinds of models, obviously, in the AI space that people might be concerned with. And I would, in my own head, thinking ahead to this conversation, I was like, of course, we've got the large language models, which output text and can reason about things. And first of all, they might just know facts that could be problematic. They can use tools. I just did a conversation with Geoffrey Irving, who's the chief scientist at the UKAC, and I had not realised before that frontier models these days are really getting quite good at shooting lab experiments from cell phone pictures. Like, they've now got to the point where You can just snap a picture of what you're working on, tell the AI that it's not working, and it will coach you through how to get it working. So that's like the know-how, the reasoning, the procedural stuff. Then we've got models that, as you alluded to with things like AlphaFold and that whole genre, are very good at making very specific predictions. What shape is this going to be? What's going to bind to what? So on and so forth. And then there's the middle ground hybrid, Things like Evo and Evo 2, where they're trained kind of like large language models on these vast datasets. In many cases, they are literal next token predictors, albeit in the DNA or the protein sequence domain. And I probably have the least intuition for those. I guess you complicate that taxonomy for me if you want, but then maybe just go through and tell me how concerned I should be in a world where there's no data controls and the models have to learn on everything we have? How concerned should I be about those different kinds of models, or possibly how they might be stitched together?

Got it. I think in general, I like this taxonomy. I was not as creative as you in terms of coming up with the different capabilities that these groups have, but I think about them largely in the same way. So LLMs, they're trained on lots of biology information from textbooks to scientific papers. They can give that information to someone that doesn't already have it. They're not already a biology expert. And that's usually called uplift. And broadly, that's a really great thing. That's LLMs teaching someone new biology, helping students learn. And really, it's a quite useful capability. There's a subset of that. subset of instances, for example, how can— how do I obtain illegally an automatic weapon? Or how do I illegally obtain smallpox samples? That's the kind of information that is clearly not something that should be provided to the general public. And that's where Frontier Labs are working to apply classifiers and refusals to make sure that that kind of knowledge is not being widely shared. Then when you think about Tools that can be used for biology tasks. I think people often call these biodesign tools. These are specialized models that are trained on biology data, and they're used to do specific things. And so I think of this as like, this is a model that gives someone a new capability. It's not about knowledge, it's about what you can do. And these types of models really require someone to already be an expert. You have to already be a computational biologist working with models to really leverage these kinds of capabilities. But the interesting thing is that it can allow those researchers to do something that was just not possible before. So before the world of protein design and AlphaFold and structure prediction, it was just not possible to take a protein sequence and then immediately through computational methods, play with new designs or try to infer its structure and function. And so those are really interesting capabilities. Again, the risks here are less about providing uplift to someone who didn't already know how to do that, but it's about giving experts the ability to do new things with biology. And then this third category, I agree, it's kind of— it's there, these models like EVO2 and ESM3, these are what I would call biology foundation models. They're trying to be general purpose in the same way that LLMs are. They're often also trained on different kinds of data. So AlphaFold obviously is trained on the PB, but it also has MSA data. And then ESM has different types of data it's trained on as well. So these are models that are often much larger Then biodesign tools, which can be small enough for an individual research group to train and host locally. Biofoundation models, they require more data, more compute. They're more expensive for groups to develop. And so you often see that these models are developed by larger organizations. AlphaFold obviously being part of Google DeepMind, not being an independent lab, independent academic lab. And then ESM is from evolutionary scale. So these types of models are trying to infer the fundamental laws of biology. They're trying to understand how biomolecular components interact across different scales and really elicit like, what are the underlying laws that govern protein function and protein structure? And in the case of Evo2, operate across different Evo2 is a model that's trained on just DNA sequence. And what they were able to show is that it can actually help with tasks across both sequence and protein and genetic regulatory circuits. So that's like operating at different scales in biology. And it's, uh, yeah, inferring laws that transfer between those different scales. So overall, lots of interesting work being done in all of these. I think the risk considerations can be separated across the different types of models. But what I would argue is that that will collapse over time because what organizations are working towards are integrated workflows. So ultimately, the dream is to be able to have your AI agent design your experiments, It will be connected to your autonomous robotics, which will conduct those experiments. The data from those experiments will be collected and then fed back into your biology foundation model, which will then be used by your agent to design future experiments and et cetera, et cetera, on loop. And these kinds of iterative feedback loops where you're getting data from the real world, I think, is where people are most hopeful for what process can kind of advance biology. Because right now we have these huge datasets that are kind of messy and collected in an observational way. But what these feedback loops would allow you to do is systematically perturb systems, systematically try to assess causality, and then use that information to get further and further improve your in silico models. The dream would be ultimately you get to a point where your in silico models perform so well that they start to replace some of the wet lab biology that you've done. And you get better and better predictions of what different drugs, for example, will do in cellular models, animal models. And then, ultimately, the dream would be that you get better predictions for clinical trials. So, you have to do fewer of those, and they have higher yields.

So, what do we want to take— I'm obsessed with, by the way, with that idea of both the closing of that loop and then also a little hobbyhorse of mine I'd be interested in your take on is this sort of latent space integration of these different modalities. Obviously, we've seen this with image and text in the sense that I can now go to a Nano Banana model or whatever and give it an image and also some text instruction, And it is understanding those in a joint way to a degree that wouldn't be possible if it was just like prompting an external image model with text, right? I could have a language model that uses an image generator as a tool. We've seen that. But this sort of deeper integration gives you just like much higher fidelity to the original, and you can do like text and image prompting in a very natural, integrated, cohesive way. So I've been wondering, I assume we're going to see it, but on what timescale do you think we see that kind of stuff in the natural sciences and specifically in biology as well, where you might say not, okay, hey, language model, you can call this protein model as a tool, but rather like, you are both. And what I want is working from this protein as an example, give me another protein that could do the same thing. and just have that all be understood in the same set of weights. What do you think the outlook is for that sort of system?

I agree that that is where the field wants to go, and it would be really useful to be able to develop that. I feel like there are a couple of bottlenecks along the way. One is the data generation piece. That is still something that requires scaling in the physical world and doing experiments in the physical world. And that itself will be bottlenecked by advances in robotics. So if we were to suddenly see robotics speeds up, we're able to do a lot more laboratory work autonomously using robotics, then the overall picture in terms of data generation also will advance. So I guess I'm hedging. I'm not giving you a timeline, perhaps next 5 to 15 years. These are the kinds of advancements we would expect.

Is that data— I mean, you said obviously we have huge amounts of just raw sequence data, but we are short on sort of the causal graph, if you will, of I did this and this resulted. Presumably a lot of that is maybe locked up at pharma companies that have done some of this stuff, or even just in the the clinical data. Do you think that if we had full access to all the data that exists, regardless of who owns it and how it was created and where it's sequestered, would we have enough data already for that kind of thing to happen? And we're essentially recreating for IP reasons something, or privacy, whatever reasons, that something we essentially do already have as a society? Or would you say no, not really, the clinical is too messy and maybe the pharma doesn't have it? I don't know.

Yeah, I would say so. Pharma definitely has lots of highly valuable data that they do not share in the public domain for reasons that are obvious, and that they are— and pharma is actually using their internal datasets to develop proprietary models in-house. They're certainly trying to do that. And if we were to suddenly wave a magic wand and say, the government says everyone has to play nice and share their datasets, How far would we get? I think that we would get, we would get a little bit further, but I think that these feedback loops and a new way of generating data, it's fundamentally a different approach. You can think of it as the existing way of approaching biology is a bit artisanal. It's a bit observational. And that results in datasets being messy, having a lot of bias and being hard to work with. And what we really need to do is shift towards a much more systematic approach where we are generating data that is comprehensive and it's systematically probing every single aspect. And that's where you really need the robotic aspect to scale that data and replicate it carefully rather than having multiple different humans trying to do the protocol. There's always differences between them. When they're collecting data. So yeah, just transitioning to a systematic approach that is enabled by robotics, I think that that is not something that you would get just by enabling data sharing across private companies.

Gotcha. Okay, well, let's pop out of that rabbit hole and come back to the main, main topic. So we've got language models that can tell people things they maybe shouldn't know. They can increasingly use all kinds of tools, including design tools. Not clear to me at this point how well they could use something like an Evo 2. But when we think about those models and their capabilities, how do you think about what capabilities should be not created in the first place? Is it about I guess probably multiple things, but yeah, maybe I'll just leave it there. What capabilities do you think that those different kinds of models should not have in order to reduce the risk to society broadly?

I think that the fundamental challenge of biology is that a lot of these capabilities would be useful on the defensive side, but it's when they're used offensively that they pose concerns. And so it's the question of what capabilities do we want, but it's also the question of what capabilities should we provide access to and how broadly and when. And so when we're thinking about advancing the future of AI for biology, I think the way I like to think about it is that we should really try to step on the gas for things that are clearly good and clearly do not have a lot of risks. And things in that bucket to me are things like virtual cell models or ways of advancing clinical trials or ways of making sure we can do better countermeasure manufacturing and distribution. So lots of things that we could do that are clearly beneficial. And then there is a bucket of things that if that capability was broadly accessible, right now, that would be quite destabilizing. So I think this is just a hypothetical example, not trying to say that this is actually the current state of capabilities, but let's say there was a breakthrough where suddenly it's very easy to use an autonomous robotic system that's quite cheap to build or get access to, and that robotic system can very quickly synthesize a pathogen. So that's like obviously a futuristic scenario, but let's say that were possible. Then that's something that probably we wouldn't want anyone to be able to buy that device off the shelf. We'd want to know who has access to that device and what they're using it for. So other things that are trending in the more concerning capability bucket would be things like viral design. And even there, there's considerations on, well, we know that gene therapy based on viruses is actually an advance in we would love to see, or other purposes for viral design, like designing bacteriophages, which are viruses that only infect bacteria. They don't infect humans. But the challenge is that when it comes to artificial intelligence, a lot of the approaches are general purpose. And so if it becomes quite easy to have an AI model that can design a bacteriophage, then it's— the question is, well, then it seems quite easy to repurpose that for pandemic human pathogens. And how many people do we actually want to have access to that kind of capability? It's probably a subset of legitimate researchers who are using that. It's not something that you would want widely accessible on the internet, especially in a world where we don't have as easily accessible countermeasures. So it really becomes an offense-dominant capability where the design and acquisition of a pathogen becomes easy and it's facilitated by AI and exists in the digital world, is being uplifted in the digital world. But then our countermeasures to a pandemic remain very physical world bottlenecked. And that's a world where it just becomes very offense dominant.

Even when, with things like a wholesale model, would I be right to worry that, and obviously one of the things you would want to do with a wholesale model is Throw stuff at it and see what happens to the cell, right? So if you had that, all kinds of great things you might be able to do. But then presumably you could also start to throw, pick your virus of choice and start evolving that in whatever way you want to, and potentially just brute force throwing the same or all these little permutations of a virus at the whole cell model. It strikes me that these things are vulnerable to like brute force attack. If they're going to be good, they're going to be vulnerable to that sort of brute force attack. Is that right? Or is there any way around that conclusion?

Yeah, you are, you are embodying the debates that people constantly have in the biosecurity community. And I think that what you're saying is correct. It is, it is, there are ways to envision every kind of biomedical advance, especially in the AI domain, there are ways of envisioning how it could be used for harm. And because of that, you have to think about how direct the harm pathway is and how consequential the ultimate harm would be. And you have to try and draw a line somewhere. So if you just compare, for example, a generative language model that had no data filtering, had no data exclusion. It was highly performative on viral genome design, and it could do that for human pandemic pathogens. There, the pathway for harm is quite direct. Someone who has very limited biology knowledge could use that model to generate thousands of potential designs, sequence them in the wet lab, see which one is the optimal candidate, and then use that candidate. So it's a fair— I mean, still a lot of work going into that, but it's a direct pathway. But what you're describing, which would then require plugging in multiple different AI models, generating candidates with a different model, then running them through a different predictive model, seeing the consequences, trying to figure out which viral candidate would cause, for example, a systemic inflammatory response or something that would target certain organ cells. So I think there is a pathway to harm there. It's just you get— you, um, when you game it out, there's more steps involved and more expertise required. And then the ultimate question you ask yourself is, what kind of actor would choose that pathway over an existing weapons pathway? So if it really requires high-level expertise that only a nation-state has access to, Is that nation state really choosing a biological weapon or are they more likely to choose something else that a nation state would have access to? And those are the kinds of questions that security professionals try to, to game out.

Okay.

Maybe let's get to the proposed solution. I've been coming at this from a ton of different angles. You've got a whole taxonomy of 5 levels of biological data. Obviously this is inspired by or pattern matched at least to the levels of security around biofacilities. So maybe just take us through the 0 to 5, like what are the kinds of data that fall into these different levels? What would the access look like? What would the precautions look like? Paint a picture of the world that you envision.

Great. I'll try and paint a somewhat visual picture. So for those that are imagining this, this is a 5-tiered system that goes from level 0 to 4. And this is modeled on what some of you may be familiar with, which is biosafety levels or BSL levels. These are the famous safety levels that biological laboratories use in terms of determining, do I have to wear a spacesuit when I go into the lab, or can I just use a fume hood to deal with my samples? And it determines different containment approaches that are required. And actually, this— the BSL system was the basis for a lot of the frontier safety policies that different frontier AI labs have in terms of the idea of having a system and your mechanisms for controlling it and kind of having it roughly on 4 tiers. But what we're proposing here is applying this not to the model and not to the physical pathogen, but rather applying it to data. And the reason that we're proposing this is because the entire biomedical research ecosystem, when it comes to AI, is built on open-source models. Academics build open-source models, they share those models openly, other researchers manipulate and change those models, and there are a lot of benefits to that whole open-source ecosystem. And those benefits are what have prevented security approaches from being applied to models. And it seems like that's going to be a pretty intractable approach. And we were looking for a different approach that could be applied to ensure that you could still still preserve this open source model ecosystem, but you're not distributing capabilities that would be particularly concerning, like viral design. So that's how we ultimately settled on biological data. Biological data, especially the kinds that I described, the more functional data, is expensive to produce, requires wet lab, and requires expertise to produce. And that's why it's a potentially useful choke point. The tiering system that we describe would preserve the vast majority of biological data as fully open access. And that's what we're calling BDL-0, where most data would be available to researchers. But as you go from levels 1, 2, 3, up to level 4, you have increasing levels of control based on how potentially concerning the data is. And the way it's essentially broken up is BDL 1 is data that would allow you to infer viral patterns. It's pretty basic security, just requiring an account and understanding who the person is and if they're a legitimate researcher. The— and as you go up, you're getting more and more focused on properties of pandemic pathogens that would really directly lead to harm. These are the properties that I mentioned to you that governments globally already pay attention to for wet lab research. So it's when you're making a pathogen more transmissible, when you are making the host range larger, if it can infect more animals, or if it can move from just infecting an animal species to also infecting humans, when you are manipulating the pathogen so that it evades the immune system. These are the kinds of properties where if your dataset has data that is link— directly linking those properties to pathogens, then it requires things like use approval. You have to— if, for example, you would go to the repository, you would say, I intend to do this kind of model development based on this data for this purpose. If you're a legitimate researcher who has a good purpose for doing this, then you will get approval versus just having this data openly accessible for anonymous access. So maybe I'll pause there.

How would you describe the magnitudes of those? Is the outer ring, the BDL0, is that like 99% of the actual raw data or how, and how small does it get when you get up to the uppermost levels?

Yeah, I would say 99% is a, is a pretty good guess. We don't actually have numbers to base this on because there isn't a comprehensive tracking system for these kinds of datasets. But my guess based on the research that we've done would be that the highest security tier, the BDL4 tier, is a very, very small subset of all data. And it would be a very few small number of specialized virology labs, for example, that would be affected. And frankly, those labs are probably already limiting access to those datasets that they produce in some way. It's just not a formalized system. So, BDL-0, again, the vast majority of data that we're talking about, petabytes and petabytes of data, vast majority being uncontrolled. And the controls that we're proposing are on a very narrow slice of data, particularly getting up to BDL-3 and 4, which I would say perhaps dozens or less laboratories affected. I am making that statement based on my knowledge of the field, but there probably needs to be some more comprehensive effort to try and figure out who exactly is generating this kind of data. And the visibility bottleneck that we currently have is what's happening in the private ecosystem. There's pretty good visibility in terms of government-funded work and less so on the private side.

So in terms of the impact that this would have, one of the things I thought was really interesting in reading the recent paper calling for these kinds of controls was the report, which I hadn't realised that there has been data holdout work done on a couple of the leading models, ESM-3 and Evo-2 specifically. Could you talk us through a little bit of what that has looked like? Put in a good word for me with Alex Rivas, please, to get him on the show. I've tried, but we did do one with Brian He on Evo. And so I have a general sense of what that looks like, but I didn't get into what was held out, how much of the overall data, how did that affect performance in areas of concern? Did it also affect— because I assume one thing people would be really worried about here is like, I don't want to have a dumb model in general, right? So if I If I slice out this data, what does that mean in terms of what can't it do that I want it to not be able to do? But also, are there things that it can't do that I would wish it still could do? Are we paying any costs with the benefits?

Yeah.

Yeah, this is an important question. I think that we all have an intuitive sense that AI model capabilities are based on the data that an AI model is trained on. It intuitively makes sense to us, but the degree to which that is true is an empirical question. And especially in biology, there is a reasonable reason to question, if I was to remove a very small subset of data, could my model just interpolate around that gap? And if your model has really internalized a fundamental understanding of the laws that govern biology, does it really matter if you start segmenting out different small pieces of data? So this was an empirical question until, as you said, some of the leading biological AI model developers actually went about doing this. And so I'll just describe the 2 examples you mentioned, ESM3, which is a generative protein design model made by Evolutionary Scale, and then Evo2, which is a generative DNA language model made by Brian He. And both of those groups had decided that they wanted to share their models, but they didn't want to disseminate the capability of others being able to use their models for viral design. So the way they went about limiting that capability was limiting what went into the training data. So in the case of— I'll just speak about Evo2, for example, given I was involved in that work, we decided to remove the sequences related to viruses that could infect humans, viruses that could infect eukaryotic organisms. But there was still some information related to other types of viruses, for example, those that could infect bacteria, that was included in the training data. And after doing the big pre-training run, We then did some evaluations where we actually checked to see, okay, is the model limited in its capabilities on certain tasks? A common task that people use in this case is looking at how well the model can do certain viral protein-related tasks and how well the model can generate sequences that correspond to functional viruses. And those were all things that the team checked and actually showed that the model capabilities were significantly less than how it did on other domains. And those evaluation results were all published as part of the EVO2 manuscript. And the interesting thing about the work that the ES— the Evolutionary Scale team did for ESM3 was that they actually had both versions of the model. This wasn't something that the EVO2 team did, but the Evolutionary Scale team had both the trained model that had been trained on all data that they had chosen to include, as well as the data-filtered model. And they were able to show a delta in performance on those same tasks with regards to, for example, viral protein function prediction. So, yeah, that's like an inkling of some of the empirical work that has been done and could be done in the future to try and suss out how much does it matter in terms of when you remove these kinds of data? What is the particular kind of data that matters? These are all questions that could probably be explored a lot more.

Could you give a sense of the order of magnitude of utility reduction? Are we talking it just can't do that stuff at all anymore, or it's in the uncanny valley somewhere? I don't have an intuition, honestly, for what I should expect.

Yeah, yeah, I can say, so for example, in the case of Evo 2, it was, it was much more along the lines of the model function was effectively random rather than just reduced by a small amount. Cool.

That's great news.

I love it when something works. Okay, maybe let me zoom out and take stock of all this stuff. So we've got a ton of new data coming online. We want to facilitate sharing. We want to facilitate all this discovery, but we've got to have some of these different classes of data controlled so that models aren't trained and disseminated on them. Presumably this also means the models themselves that are trained by the ESM team and the Evo team also need to be controlled. I guess we shouldn't be expecting government if we don't have government controls on actual wet lab gain of function research. It doesn't sound like we're going to get government control on this sort of thing. So what are we doing? We're campaigning and trying to build private agreement and consensus. Is that the play? And how's it going?

Yeah, great questions. Just to take a step back in terms of what happens with the model and should the models be controlled, I think in the case of EVO2 and ESM, they had effectively neutered the concerning capability from their model. And so that's why they felt more confident in disseminating those models. And I think, yeah, ultimately, it made sense to be able to share those models openly because they had worked to reduce the concerning capability. But what I was more so referring to is if you do end up implementing data controls as we propose and you get access to BDL-3 or BDL-4 data for the purposes of training a model, then what you wouldn't want to happen is that then that model is then shared publicly because effectively then your mitigation didn't really do anything. And we do make sure to mention that any AI model trained on that kind of secure data should also be shared in a secure way if it needs to be shared. But yeah, in terms of what should be done next, I think that the interesting thing about data controls is that there's a lot of policy precedent. We already do this in other domains. We do it for privacy-related data. We do it for human genomics data. And so there's reasonable precedent for extending that same approach to data controls. So I guess my collaborators and colleagues who worked on this are optimistic that perhaps this could get picked up by policymakers. But of course, It's a new concept, and so we'll keep plugging away at it. And then more broadly, in terms of what will happen with wet lab gain-of-function work and model capabilities, I think that actually both the Republicans and the Democrats have decided that this is a bipartisan issue. And it was under both the prior Biden administration and the Trump administration that some work on advancing regulation on wet lab gain-of-function work went ahead. So I was really glad to see that progress. And I think that also both the US Casey and UK AC are both doing really great work with regards to biosecurity and biological model capabilities. I think that US Casey, just speaking given that I'm more familiar with their recent work, they have put out RFIs, requests for information, from the scientific community to better understand Okay, we know that developers are doing some of these data filtering steps. They're trying to, to limit model capabilities. Maybe we should have a more systematic approach to figuring out what are all the capabilities that we should be concerned about? How can we test for those capabilities? How do we effectively mitigate them? And so I would just love to see Casey be resourced and staffed to be able to advance that line of work, because I think a lot of developers are trying to do it themselves in an ad hoc way. but they don't have the same security access and intelligence access that something like USKC has.

Do you envision a sort of centralised— like, I'm still a little bit fuzzy on the workflow of something like this. If I, for whatever reason, am out here generating some sensitive data that maps viral sequence onto Viral capability. And now I'm like, okay, I've got this data. I want to be a good citizen.

What do I do?

Are you envisioning a scenario where I take it to a central databank, give it to them, and then scrub it from my computers and go on with my life? Because we can't have— everybody can't be doing these sort of security levels for themselves, right? So there would have to be some, maybe not totally centralised, but at least no more than— countable, relatively small countable number of the organisations, entities, or whatever that would have custody of the data. So is that kind of the idea that people would feed it in and then they would be expected to actually delete from their own servers? Is that realistic to expect people would do that? How do you expect that to actually go?

Yeah, this is all about how do you operationalise this system that we're proposing. And the way that this has been done in the past is through things called trusted research environments, or TREs. And OpenSAFELY, the system that I mentioned at the beginning, is one of these. And so the idea would be that you have a secure environment that hosts the data and also enables researchers to bring their code to the data and answer questions. And obviously, in the age of AI, the ideal would be that you have compute resources available to that, although we'll see if that would be possible. But I think the way that you would set this up is that, yeah, we considered different approaches. One approach could be that the government sets this up. I think that after we looked into this, it seemed less ideal. I think a lot of researchers have been unhappy with some of the trusted research environments that governments set up. And I think that there are private actors that could probably do a better job of setting up really savvy trusted research environments that work well for researchers. And we have some examples of this that I'd mentioned and that are included in the paper. So what we suggest is that institutions, not individual researchers, but institutions like universities or larger research collaborations, set up a trusted research environment if they want to do this kind of data generation. And they would be given standards that the government would set, but they would be in charge of actually doing the building and maintaining just because they're probably going to be better at doing that. The interesting thing about a trusted research environment is that you could actually imagine this also being a boon for researchers. Because when you're trying to do AI research, what you want is a centralized, integrated platform where all the data sits and you have access to it all at once. It's much less useful to have individual datasets housed with individual researchers. So if we really wanted to step on the gas of advancing countermeasures research for pandemic pathogens, maybe this is something we would want anyways. We would want an integrated system that hosts all the data and really makes it easy for our researchers to interact with. What we're proposing to layer on top of that is that you do have some security that goes along with that system. So not only are you hopefully getting the benefits of the integrated platform, but then you're also getting the security controls given the sensitive nature of the data. Gotcha.

When it comes to like monitors, I know that you had alluded to some rare points of cross-administration agreement. One of those I understand is the insistence, or I don't know if it's fully a requirement, but I think it's verging on requirement, that DNA synthesis companies apply certain classifiers or whatever to try to detect if somebody is trying to get a harmful sequence synthesised through their company. There's also things like wastewater monitoring and I wonder if— I guess I have a couple of questions on this. One is, is there an equivalent of wastewater monitoring for data? Would it make sense? I don't know if the data of concern is the kind of thing that could be identified if it's just hanging out on somebody's lab website. Potentially, people might not even fully realise if they've generated problematic data. So is there any sort system you could imagine to go around and identify data that's out there and then classify it as possibly harmful? Or is that so difficult of a classification problem that it would be doomed from the start?

It's a really interesting concept. I have not thought about it too much, to be honest, but I feel like you would probably need— there's 2 approaches to figuring out What the data landscape is for this kind of data. One is a more active contribution-based approach where the government says, if you think you're creating this data, you have to actively contribute it to these repositories. What you're describing is a passive approach where no one has to take any individual action. Some system is flagging the data and then it perhaps even like automatically collects it and then scrubs it from its original source. That would be the dream. I'm not aware of any system like that. I think it would probably take a lot of work to do. I think that the default approach has been the first one, which is voluntary or like active contribution by the researchers who are generating the data. But what you're describing would be cool and would be the analogy to wastewater surveillance, essentially.

When it comes to just the quality of monitors in general, a big thing I always think about in the in the kind of rest of the world as it pertains to AI is we're in this sort of weird in-between phase where things are coming online, but the world hasn't really reacted that much yet. So, for example, AI agents are coming online. There's a lot of concerns around things like prompt injection and whatever, but the world hasn't really become like a very dangerous place for an AI agent so far, right? Not that many people have set up honey traps or prompt injection attacks to try to throw my OpenCLaw off its path and talk it into doing something that they want it to do. I assume that's going to happen a lot more and there's going to be an arms race, techniques to prevent my OpenCLaw from falling for it, and ever more sophisticated jailbreaks, whatever. I assume there's got to be something analogous in the biological domain where, for starters, these classifiers that the DNA synthesis companies are running, I'm guessing that nobody really yet has tried to evade them. I wonder if you see that kind of dynamic developing on the horizon. And Nicholas Carlini is a past guest who emphasises, like, the attacker usually gets to act last, the defences are set up, and then the attacker always has that advantage of knowing what they're up against. Maybe not always, but often. Do you see any of those kind of dynamics Now, or do you worry about them in the future? And I guess if you extrapolate them out, do you see us getting to a place where we have a high level of confidence that we're in a defence dominant world and we're going to be able to keep all this stuff under control? Or is that unto itself still like a very open question for you?

Yeah, I have some ideas, but maybe let's start with the gene synthesis screening and the different— so there's a few different attack surfaces. I'll just describe them as that. The first one being gene synthesis providers. So what you're describing is the system that is currently voluntary, where gene synthesis providers, companies that make pieces of DNA and sell them to researchers, they have implemented systems where they are checking to make sure that someone just didn't just order pieces of smallpox, someone didn't order pieces species of Ebola. And they have— it's a twofold mechanism. The first is an automated mechanism that is just essentially sequence matching. It's looking, does the order match a sequence from Ebola? Does it match a sequence from smallpox? If the automated system thinks that there is some degree of concern there, it gets kicked up to a physical— I mean, a human researcher, a human expert who then looks at that and determines, is this something of concern or not? And also what's happening in parallel is some degree of KYC, know your customer. So who did the order come from? Is it a researcher? Does this researcher work with these kinds of pathogens all the time? Have we spoken to them before and this issue has come up before? Those are all the kinds of questions that are being addressed as part of gene synthesis screening. Gene synthesis screening is something that 80% of companies already implement. And over the past years, it's become much more cost-effective. Effective to do. So initially, it was a bit of a cost barrier. They really did a lot of work to make this a cost-effective system, because you can imagine that if this is something that's being run on every single order that's coming in, it has to be cheap enough to do. Otherwise, it's really burdensome for these companies. And so currently, it's a voluntary system. Like I said, the vast majority of companies already do it. But the concern is, if I'm a bad actor, then a voluntary system that 80% of companies use isn't really going to stop me from obtaining sequences that I want to get my hands on because I'll simply just go to the companies that don't do this screening. And so what's now being advanced is the idea of making this a mandatory rule that all companies have to follow. And that would really limit the access that someone has to physical specimens that they might try to use to turn into an infectious pathogen. But I guess in this world of where is the research going, what are the capabilities that people want to achieve. The dream of where future— the future of biology is going is that I, as a biologist, no longer even have to step foot in the lab. I have my autonomous cloud lab that I can fully control. I'm sitting at home using cloud code to help me, and I can just program some designs to be done on certain pathogens. To be completely frank, like, we're nowhere close to this world. It's still going to take a lot of work. Right now, the cloud labs that you hear about still require a lot of human input. It might not be specialized biologist input, but there is still a person picking up samples from one bench and moving it to the other, and that's a real bottleneck. But in a world where you do have a fully remote, highly sophisticated cloud lab, then you can imagine if there are agents, you know, if I'm a bad actor and I have an army of 1,000 agents that are just trying to hack their way into this cloud lab. You want to make sure that if your cloud lab has sophisticated capabilities related to pathogen creation and design, you need some cybersecurity around that. And so that's like a much more future-oriented thing, but something you could imagine becomes applicable later. And right now there's a fundamental kind of information infrastructure layer that we're missing. So both for cloud labs and for gene synthesis screening. If I, as a bad actor, try to obtain sequences of Ebola from one company, small pieces from one company, a few pieces from a different company, and I split up my order across different companies, there isn't some kind of system where all those companies are easily communicating that information to each other and checking those orders against each other in real time. This is something that's bottlenecked by information sharing between companies. That's like also kind of a similar concern to why the Frontier Model Forum was created in terms of you want private companies to be sharing security-related information. You need a legal infrastructure for that. And yeah, this is like, who would house that? Would that be the FBI? Who's facilitating this? These are all kinds of policy questions that need to be addressed. But yet perhaps we can end on a more optimistic note. I'm happy to give my vision of like what a defense-dominant world looks like and see what you think of it. I think that overall I break this up into 4 broad buckets of interventions. Well, perhaps to take a step back, I think people often think about what's the theory of victory here? How do we have a single unified strategy like our strategy for nuclear deterrence? Like, we have a single unified theory of victory for nuclear deterrence that has served us well for decades. How do we get there for biology? And I guess after having thought about this for some time, it just feels like biology is very different. It's a distributed technology. It's dual use. You want to give a lot of people access to it. You're trying to limit a subset. And so there's all these aspects that really make a unified theory of victory seem much harder to get to. And so I think the most successful approach is likely to be defense in depth, a layered approach, multiple different defensive strategies applied at once. And the 4 buckets that I divide this up into are deter— I'm sorry, I should start with delay, deter, detect, and defend. So delay is essentially limiting access to concerning capabilities. Gene synthesis screening would fall into that bucket. You're delaying the dissemination of that capability to get access to DNA fragments, for example. You could also imagine what we're describing for our data controls as part of delay. Then there's deterrence. Deterrence is figuring out how you can punish someone for using a biological weapon. We do live in a world where there's an international treaty against biological weapons. I'm glad that we live in a world that has that treaty versus not, even if overall that treaty is, is on the weaker side in terms of the actual mechanisms we have to ensure people are complying with it. But that, that's one thing. I think where deterrence breaks down is if your actor is not rational and doesn't respond to typical punishment mechanisms. So that's the challenge. But then, then in terms of detection, this is also something we talked about, which is how do we have a distributed passive surveillance system like our radar system for ICBMs that will just detect when there's a new pathogen without us having to go out there and look for it, perhaps even for pathogens where there are no symptoms. When HIV was spreading early on, it would've been amazing to have known that much earlier. And that's particularly important for pathogens that take a long time for patients to develop symptoms. So having some kind of global surveillance system, people often refer to this as bio radar or bio threat radar. It's not something we currently have. And then the last pillar is defenses. Really, what are our defenses once there is a pathogen online already? And I think that people often think of defenses as things like vaccines and countermeasures, but I would encourage folks to be much broader in their. in what they envision defenses to be. Because I would argue that I'm sitting in my home right now and I actually have defenses all around me. I'm drinking water that's been centrally filtered. I know there's no cholera, no pathogens in that water. I have screens on my windows, mosquitoes can't get through them. I know that I'm not gonna get malaria if there was malaria outside. And so there's already a lot of kind of public health defenses built into our environment, but we don't have that for airborne transmission. And so one thing that's being explored by organizations, one that comes to mind is Blueprint Biosecurity, they are exploring built environment defenses to sterilize the air. This is using approaches like far UV and other approaches like glycol vapors. So could you passively sterilize the air so you don't even have to detect the pathogen, you don't even need a vaccine. You kind of just always know that you have passive protection around you. So hopefully I would say like, that's a pretty comprehensive approach. If we managed to do all of that, it would take a lot of work and investment to get there, but it would make us a lot safer. Cool.

Shout out briefly to the AeroLamp folks for my son's hospital room, because he was obviously going to be so immunocompromised much of the time. We thought, what could we do? It's cancer is obviously the biggest concern, but infection when you're so immunocompromised is like another thing that poses a real risk to kids in that position. So between HEPA filters that we bought and installed around the house and the AeroLamp dev kit, which we have mounted above his hospital bed every time he's been in the hospital, we've hopefully taken some of the risk off the table of him getting any kind of infection while he's going through all this. But yeah, well, that's a great vision. I hope we implement it. You are doing God's work for spending your precious time and energy on this. Anything else we didn't touch on or any other calls to action or ways that people can help you that you would want to leave people with before we break?

I think we covered everything and we really appreciate your interest. It sounds like you're one step ahead of everyone else in terms of already getting your kid outfitted with all the defences he needs. So Yeah, it's great to chat with you.

Jassy Panoo, thank you for being part of the Cognitive Revolution.

Just sitting there, still viable, still patient, in a room they'd half forgot. Why we spent $11 billion on a radar for a shot that hasn't come from Russia since we learned to hold our breath. But a cough in the market, well, we'll get around to that, I guess. 5 more That's the distance from a bird flu to your door. They published the directions. So would you think they'd publish 4 or 5 mutations in a protocol? Sitting free and open wide in the system that's supposed to catch it. Unless the devil slipped aside. Some man stitched a cousin of the smallpox back together, wrote it up step by step, pinned it to the public record. No law broken, no hands slapped, just another paper file in the library of everything that could kill every living child. They say the data's nice and neat. I'm not sure I believe that. 99%, just regular and fine. But the last percent's a hell of a gamble on the whole of humankind. 5 mutations, that's the distance from the bird flu to your door. They published the directions. Oh, what'd you think they'd publish for? 5 mutations, that a protocol. Setting free an Open wide in the system that's supposed to catch it, lets the devil slip beside. Yeah, now the machines are reading faster than the people write, and they're learning things on Tuesday that we won't know until Friday night. One of them went looking for It wasn't supposed to find, crack the lock, read the code, route it right around the blind. So tell me, who's the watchman on the data tonight? Who's standing at the door between the knowledge and the blind? We built radar for the rockets and a fence for the cattle. Got a filter for the water. Got an alarm for every alarm. The smallest thing that travels on the air between your lungs. Got no siren, got no signal, got no warning when it comes. There's a woman Coming down at Hopkins, says it doesn't have to be this way. Lock the dangerous, free the useful, put the liars on display. Delay, detect, defend. She says it like she's tired of saying it twice. I wonder what it cost us. I wonder what's the price of knowing what we know and doing nothing with it. I wonder what's the price of knowing what we know and doing nothing with it. To knowing, I wonder what it takes this time. I wonder what it takes to care while the mutations keep on going. And the protocol's still there.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitive revolution dot ai or. Or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.

Thank you.