Scientist Stories: Brian Hie, Evolution of Human Antibodies From General Protein Language Models

Hi everyone, please join me in welcoming Brian, who is currently a Stanford Science Fellow in the Stanford University School of Medicine and a visiting researcher at Meta AI, where he works on algorithms for protein evolution and engineering applications. He received his PhD in electrical engineering and computer science from MIT, where he focused on computational biology, including methods for understanding viral evolution. Single-cell transcriptomics, and small molecule activity. Welcome, Brian.

Well, thanks, Andrea. First off, I'm a big fan of this seminar, so it's like, it's an honor to present instead of just being in the audience. All right, cool. So yeah, to get started, my name is Brian He. Today I'll be talking about methods using this class of algorithms called protein language models in order to enable more efficient evolution of human antibodies. And this is joint work with some wonderful co-authors from the Peter Kim lab at Stanford University. So to begin, this project started in part as a inspired by machine learning guided directed evolution. So in traditional directed evolution, you'd have multiple rounds of mutation and then experimental measurement, and then you keep the mutations that improve fitness as determined by the experimental measurement. And in doing so, you conduct an evolutionary process in the laboratory. And then more recently, There's been a lot of work in accelerating this process using a predictor or a surrogate model that takes in a sequence and predicts some fitness value. And because it's much easier to evaluate a computational model rather than collect the experimental data in the wet lab, this predictor can help accelerate machine learning-guided directed evolution. So this project kind of in part was motivated by, you know, if I have like friends that come to me and they ask, you know, I want to evolve this protein of interest, are there any ways that I can do this? And oftentimes, unless you're studying some of the most very well-studied proteins, you won't actually have a lot of or any task-specific training data at all with which to train this surrogate model. And more generally, in the most general setting, which is also probably the most common, you just start off with a single wild-type sequence and you don't even know the structure or anything like that. You just have a sequence and some experimental method for, you know, assaying that sequence for the desired fitness property. And so, you know, still in most experimental settings especially, what you would do in this case is just collect initial training data or just collect, you know, in a high-throughput way, assay random mutations to the wild type or do brute force deep mutational scan. But that seems like we can do better than random guessing and or brute force search? So this is the question that started this project off. And so what can we do when you don't have any experimental measurements to train a surrogate model? So the idea underlying this specific project is to use past evolution to guide artificial evolution. And why is this like a good inductive hypothesis? So, you know, much has been said about the combinatorially large theoretical space of proteins. So you can think of like all possible antibodies. The number of possible antibody sequences is larger than, you know, the number of atoms in the universe, as an offhanded statistic. But, um, so this space seems intractably huge, but actually when we look at natural evolution, it seems to be exploring a much, much smaller protein space designs. So for antibodies, for example, it's now been shown across multiple pathogens that 2 individuals that are completely different will evolve very similar, in many cases will evolve very similar antibodies to the same antigen. So this is a very recent example. These are 2 antibodies. One is this antibody from Eli Lilly, Bebtelovimab. The other, this antibody S2X324, which was isolated by the Vista Lab. They have 87% amino acid sequence similarity in the heavy chain, 91% similarity in the light chain. They both bind the same epitope and have very similar bound conformations. And so this is one example of, like, you know, this convergent evolution of antibodies. Another example from viruses, I'll, you know, keep this a little bit more brief, but when people were studying 2009 pandemic influenza, like the swine flu, they were noticing that, you know, this, this thing actually looks a lot like the 1918 pandemic virus. And actually, there's, you know, 80% amino acid sequence similarity in the antigenic regions of this protein despite, you know, more than a century of antigenic drift. So, you know, the takeaway is that, you know, despite this extremely large theoretical space of proteins, you know, perhaps natural evolution is utilizing a much, much narrower search space. And as a protein engineer, we can potentially take advantage of that to our benefit as well. Okay, so if we want to use past evolution to guide artificial evolution, how do we kind of learn properties, learn evolutionary patterns? So the class of algorithms that we used in this project are called protein language models. I think this is a machine learning for protein seminar, so I won't go too in-depth into the different types of language models and assume some general sense of familiarity with them. But at a high level, a language model will define some probability distribution over sequences, in this case, protein sequences. One very common type of language model, a way of training these language models is to say mask out some residues in a sequence and try to predict them in the output. And so these masked language models say, you'll take some sequence context and then output a likelihood over all amino acids for what belongs in this sequence context. There's other types of language models, autoregressive language models as well. And the way you train these language models, in particular a class of language models that are more general, is to just train a very large language model over all the proteins in UniProt, for example. So these are just generic protein sequence corpuses. And in a result that I find, I still find quite fascinating and still surprises me a lot is that if you just look at the likelihoods that are learned by this language model, when you train them over these large sequence corpuses, is that they'll have in many cases fairly high correlation with experimentally determined measurements of fitness. And these are highly specific laboratory-defined measurements of fitness. So for example, you know, beta-lactamase, you look at ampicillin resistance, for example. But in many cases across many natural proteins, these likelihoods will just, without any task-specific training information as to, you know, that specific fitness definition, will just have, you know, good correlation with those experimental measurements obtained through some high-throughput assay. And in some cases, you know, on par with training just an epistatic model directly on variation within the sequence family. There's also been— there's been a lot of work following this. So this is a language model, ESM-1B, from a team at Facebook AI Research. The same team also has a model, ESM1v, that outperforms ESM1b at the mutation effect prediction tasks. There's also other more recent models. There's CART from a group at Microsoft Research, and there's really actually really interesting work, the Transeption model out of the groups of Debra Marks and Yaron Gall, that appears to do really well at variant prediction. Even though it's trained on a generic sequence corpus at doing this variant effect prediction. But to me, it's still very interesting that this even works in the first place. What we've also seen in previous work is that if you look at natural evolutionary trajectories over, in some cases, decades for viral evolution and geologic eons for ancient evolution, these trajectories will actually progress in the same direction as higher relative language model likelihood. So another, you know, the takeaway is that there's some evolutionary information that's being learned by these language models. So, you know, the big question is, you know, can these rules that are learned on past evolution help guide evolution in the laboratory? So there's a bit going on in this cartoon, but fundamentally, these language models, if we're going to use them to evolve, say, an antibody for higher binding affinity, these language models actually are really quite naive. They see an antibody sequence and they have no idea that the antibody binds to anything. Let alone a specific epitope of a specific antigen. And so we're calling that specific laboratory-defined selection pressure extrinsic fitness. And so if you want a language model, for example, to help predict this extrinsic fitness, this specific selection pressure, it will need to do so indirectly. So it can't— there's no way that it could predict really directly predict extrinsic fitness, at least explicitly. So I'll need to go through what we're calling intrinsic fitness, and this is related to sequence intrinsic properties that can be learned from sequence variation alone. And this is also related to ideas of folding stability or thermo or entropic stability and also evolutionary plausibility. There's potentially non-uniform mutational biases in the sequence dataset. So, you know, we're thinking of this as maybe what makes a valid antibody. And, you know, if you can, if these language models are intrinsic fitness, is like a good prior for extrinsic fitness, then you could just acquire based on high intrinsic, sequence intrinsic fitness or properties, and naturally enrich for high extrinsic fitness. There's another scenario in which you're just looking at high intrinsic fitness and that specific selection pressure, high extrinsic fitness, is so rare. This is not a good prior for— intrinsic fitness is not a good prior for extrinsic fitness. You won't be able to enrich substantially for it. The pros of using just a sequence-intrinsic model for guiding evolution, I think, are pretty clear. You don't need to explicitly model p of extrinsic fitness. You don't need to collect a bunch of task-specific data and train a supervised model on on that information. And because we're using sequence-intrinsic models, you don't need protein structure as well, and we're also jumping directly to extrinsic fitness, so you actually don't need to know the biological mechanism by which a change in the amino acid sequence will affect that phenotype. So this is actually a very, very general setting. You would just need a wild-type sequence and some assay for measuring extrinsic fitness. The cons are also pretty obvious, I think. There's not really— it's not guaranteed that high intrinsic fitness, or what's learned by language model, would be a good prior for the actual selection pressure that you want to optimize. In this case, binding affinity. You could just be acquiring high-likelihood mutations from the language model, and these could potentially have nothing to do— or a very rare subset would still improve, say, binding affinity to a particular antigen. So these are the pros and cons of this approach. So, you know, I guess I was kind of maybe naive enough to try this out, and we're actually going to do this experiment where We just measure extrinsic fitness of the highest likelihood language model mutations. And so we're doing this in the context of affinity maturation of human antibodies. So this is a very common application of directed evolution. A lot of therapies are based on human antibodies with high affinity for a target. And so this is the setting that, the main setting that we're using to, test whether or not intrinsic fitness is a good prior for extrinsic fitness. We require only a single wild-type sequence, in this case the sequence of the variable region of the heavy chain, the VH, or the variable region sequence of the light chain, the VL. The way we're applying these models is actually quite simple. So we're just computing language model likelihood for all possible single residue substitutions And we're just testing those with higher likelihood than wild type. There's a little bit more details. So basically we're doing this evolutionary experiment in a very low throughput manner. So this is like the most— pretty much the most low throughput. I'm just hand expressing all of these antibodies, no multichanneling or medium throughput methods. Or anything like that. So to further narrow the substitutions for testing, we use this consensus scheme. So there's multiple language models that we're considering: ESM1B, which is trained on UniRef50. It's a UniProt downsampled to 50% sequence similarity. And ESM1B, which is trained on UniRef90, which is the same thing but downsampled to 90% sequence similarity. So these are generic protein sequence corpuses described in these references. And we're prioritizing these substitutions with higher likelihood than wild type that are recommended by multiple language models as a way to further narrow that space. And we also prioritize mutations that are the highest likelihood at a given site. And this gets us down to around 10 or so variants— or 10 or so single residue substitutions And then we have this multi-step evolutionary process. So in round 1, we're just measuring those substitutions. We're just making a variant with a single residue substitution so we know, you know, its variant effects. And then in round 2, we're just recombining. In some cases, we combine all possible combinations of the best performing substitutions, and others we manually recombine them. Because the space is just too large. And in total, we measure 20 or fewer total variants across both rounds of evolution. So this is, you know, very, very low throughput, a very, very low throughput evolutionary process. And we picked 7 antibodies that we wanted to affinity mature. These are chosen based on, you know, relevance to things that the chem lab is interested in, which is viruses, and as well as, you know, a diversity in antigens, a diversity in degrees of maturity. So MEDI-8852 is this highly matured antibody that was isolated from— its parent was isolated from a patient and then optimized in order to bind different phylogenetic groups of influenza. So there's two main groups, Group 1 and Group 2. It's very hard to bind— to have an antibody that binds to antigens from both. And so this antibody went through a lot of in vitro evolution in order to expand its breadth. We look at the germline inferred sequence of MEDI-852 as a kind of unmatured version of this antibody, the unmutated common ancestor, basically kind of trying to rewind the evolutionary clock. And seeing how far we could get from its, you know, the germline precursor of this antibody. And this antibody only neutralizes Group 1 influenzas, so we were particularly interested in seeing if we could get breadth to Group 2 influenzas. MAV114 and its UCA, similar to MEDI, MAV114 is, you know, an FDA-approved therapeutic for Ebola. And then we have some SARS-CoV-2 antibodies too, which, you know, had emergency use authorization for some of the earlier variants of SARS-CoV-2. And C143, which is an unmatured SARS-CoV-2 antibody that's patient-derived, not computationally germline-inferred. And I think— I don't think the structure for C143 is even available as well. Which is part of the reason why we chose it. And so these are the 7 antibodies we're considering. So first I'll just look at the MEDI8852 trajectories. There's a lot going on in this figure, but I'll walk you through it kind of step by step. So this strip plot shows the fold change in KD, or the measurement of binding affinity, in the first round of evolution. And you can see that when we just make those single residue substitutions to the wild-type sequence and we just test them for binding affinity, even though the model had no idea that this bound to, in this case, H4 Hubei influenza hemagglutinin, glutinin, we saw that 2 of those mutations had a significant improvement in affinity. And then in the second round, we just combined these 2 mutations and showed another variant with, you know, even more improved affinity in round 2. And, you know, excitingly, the Medi does well across different strains of influenza, but it doesn't do so well on H7, but we are improving the breadth of binding to H7 as much as, you know, 7-fold for this one H7 antigen. For the UCA, we have, you know, 3 variants with a single substitution from wild type that are improving affinity, maybe a 4th one. And so we recombine all possible combinations of these 4 substitutions in round 2 And, you know, thankfully the improvement in affinity seems to be on average much higher than in round 1. And what's also interesting is that we also, we appear to be, even though we're screening against, you know, group 1 influenza hemagglutinin, this antibody also appears to be acquiring some breadth of binding to group 2 antigens like H4 and H7. As much as a 5.4-fold improvement in affinity for H7 and a 23-fold improvement in affinity for H4. And so this is, this is very promising. These mutations were just recommended by a model that had no idea what hemagglutinin— what, well, that these sequences bind hemagglutinin. They had no idea that these formed any kind of protein-protein interaction, so is quite promising. We also measured thermostability in terms of melting temperature, and interestingly, for MEDI, it seems to be about the same. For the UCA, interestingly, it might be taking a little bit of a hit in thermostability, but it still melts at a high temperature. So, you know, higher affinity for a lower thermostability. The remaining trajectories I'll walk through a little bit faster. They're similar to— laid out similarly to MEDI. So, you know, for MAB114, we get up to, you know, a 3.4-fold improvement of this antibody against the glycoprotein of Ebola virus. And in this case, maybe the thermostability is also improving by a little bit. For the UCA, some of the mutations in combination exhibit, like, the super synergy, and we think that is mostly mediated by this really stark improvement in thermostability from 74 to 82 degrees, and that results in, you know, up to 160-fold improvement in affinity against the protein. So it goes from It also goes from, you know, we test neutralization against pseudoviruses with these antigens kind of decorated on the surface, and interestingly, this antibody goes from just not neutralizing at all in its UCA wild type to exhibiting some neutralization in the affinity matured version, which we think is mediated by improved thermostability. The highly matured, you know, clinically relevant SARS-CoV-2 antibodies, we do get some improvement in might be a little bit more modest than MEDI and MAB14, but in some cases still significant and comparable to a mutation introduced in a clinically— so this citrovimab is a clinically available variant of S309, which is the patient-derived antibody that we also evolved off of. And so we get comparable, maybe a little bit better improvements in affinity and better thermostability than citrovimab. Interestingly, C143, we were able to rescue its binding affinity to a variant of SARS-CoV-2, the beta variant, which, you know, at the time was the, you know, the most immune-evasive variant of SARS-CoV-2 prior to Omicron. Then Omicron kind of just made the affinity bad, but, you know, maybe we're still a little bit better. But I thought that was also a cool result that at least for the beta variant we were Basically able to rescue the binding affinity with some of our language model recommended substitutions. And I alluded to this before, but we also test pseudovirus neutralization. So we set up an infection of different cell lines and measure viral infectivity in the presence of these antibodies and show that binding is pretty well correlated with improvements in neutralization as well. I should also mention this data is not in the preprint describing the paper, but we have been getting questions about it, so I just thought I'd briefly mention that, you know, are these improvements in affinity just driven by promiscuous binding? So we can test for what's called polyspecificity. So this is, you know, just the ability in this assay to bind like cellular junk. So in this case, this is all soluble membrane proteins, and we use this— the polyspecificity assay described in these two references, basically testing for binding to soluble membrane proteins via, you know, flow cytometry. It's, you know, biotinylated soluble membrane proteins, and there's a streptavidin fluorophore. There's these controls. Olatuzumab is a clinical antibody with very low polyspecificity. Ixekizumab is a clinical antibody with high polyspecificity. And then 4ATN is a research antibody with extremely high polyspecificity, and its polyspecificity is part of the reason why it was not developed clinically. And so in all cases, we are— the wild-type antibodies seem to be between the two clinical— the range of clinically acceptable polyspecificity. And our affinity-matured variants don't appear to be changing the polyspecificity drastically, at least significantly so. Maybe, you know, these Regeneron variants are actually improving specificity and maybe, you know, maybe a little bit an increase here, but not significantly so. And yeah, so ultimately we are screening these variants against, you know, complex antigens. So if it's acquiring specificity to, you know, complex antigen, and hopefully that also translates into better what's called polyspecificity. Just very quickly, you know, these are some results compared to, you know, other studies that are doing affinity maturation of antibodies using various techniques. And these are all really great studies. They're all extremely cool in their technologies. So Mason et al., they have this mammalian cell surface display approach where they can screen up to 39,000 sequences that they make to train a supervised model. Shan et al. do this really cool geometric deep learning to predict binding affinity based on the antibody-antigen complex. There's this very cool continuous evolution method where you can continuously evolve, in this case, a nanobody against— they did it against SARS-CoV-2 RBD. And in all cases, these, you know, full changes in affinity, so 2.9-fold 2.1 or 5.5-fold, and then 2.3 to 580-fold. Now it's very cool that, you know, we're in the range of all of these technologies, but each of these technologies involves, you know, a very highly resource-intensive component. You know, making a mammalian cell surface display library to train a model, or, you know, sorting cells based on that are continuously evolving, or sometimes the complex— that requires structure determination, and sometimes that might not even be available. So I thought it was a very promising signal at least that we're getting a similar— within the same range of fold improvements in KD as these other studies. Of course, these are not directly comparable, but still it's a very promising signal. I've also— we've also been getting questions about, you know, what happens if you do have more resources that you would want to spend on affinity maturation or on evolution. And so this technique can be used as, you know, a tool alongside a lot of other technologies. So, you know, if time is not an issue, you can definitely mix this with other approaches So you can supervise a model with additional rounds of directed evolution potentially from that initial zero-shot or that initial language model recommended set. Something that we've actually kind of tried to determine the structure of some of these complexes. So if you want to use a structural method, sometimes the initial complex might not have high enough affinity to determine the structure. So you could imagine, for example, using a method like this to improve the affinity, then determine the structure, and then do, you know, rational engineering on top of that. As an example, if high-throughput screening is not an issue, so if you have an experimental budget for, you know, some 100,000 antibodies or something, and an approach like this decreases your search space by, you know, 100-fold or something, then you can potentially screen 100 more antibodies using the same experimental budget. Or something that we really wanted— the two rounds was kind of trying to do— is the language model will be recommending these substitutions. If you have a budget for high-throughput screening, then maybe you just directly proceed to combinations and just make a combinatorial library of the language model recommended changes. And another thing, if money or resources is not an issue, then you could potentially have a more resource-intensive readout. So we use, you know, biolayer interferometry to test binding, but if you're only testing 10 to 20 variants in total, then maybe you can proceed to something more resource-intensive like live virus neutralization. And there's actually someone in the Kim lab that's been having success directly using pseudovirus neutralization as the readout to evolve. Yeah, so the main takeaway is that this approach will be like one part of the toolkit and hopefully will increase the downstream efficiency of evolution moving forward. Okay, so fundamentally we were very, you know, surprised that this method was working at all because these language models have no idea that these antibody sequences are binding to anything. And so, you know, our current running hypothesis as to why this is working so well is that the information that's being learned from just this sequence, the sequence intrinsic information, is in this efficient scenario, where it's just enriching for high extrinsic fitness mutations. And so even though we're not directly modeling binding to hemagglutinin, we're still— the thing that we are modeling, just the sequence, basically sequence recovery or the sequence likelihoods that are learned by the language model are strong enough prior for binding to hemagglutinin. So we kind of illustrate this in a cartoon. You know, maybe you're a ball and we have, you know, two definitions of fitness here: extrinsic fitness and intrinsic fitness, where intrinsic fitness makes up this rainbow road. And if you're just random guessing, you'll just fall off the intrinsic fitness manifold very easily. But if you have a way to stay on this intrinsic fitness manifold and because it's a good prior for extrinsic fitness, then you can improve the efficiency of evolution. So we refer to this as the efficient manifold hypothesis. Just, we're in a setting in which intrinsic fitness is like an efficient prior for extrinsic fitness. And one thing I wanted to mention, and this result is very widely known in the machine learning community but potentially not so outside of it is that we're using models that can predict mutational effects beyond antibodies. So these models were not specialized to antibodies, and so this is an example from this CARP work from the group from Microsoft Research showing good Spearman correlation of these general protein language models across a number of deep mutational scanning datasets with very specific definitions of fitness, the transeption work as well. I think they expanded quite a bit the number of mutational scanning datasets used for benchmarking. And we've also shown that if, in addition to Spearman correlation, if you just look at the If you just did an experiment like we did for antibodies, so you're in this very, very low-end, high-efficiency, or a very, very low-throughput regime, would a language model be in this efficient scenario and give you some enrichment over, say, just random guessing? It turns out in that a number of these proteins, using deep mutational scanning datasets as held out like validation, you would substantially or significantly enrich for high fitness mutations using the same exact model that we used to affinity mature antibodies, but in these other very, very different definitions of fitness as well. So this is a— we have a preprint describing this work, and this is the reference here. And yeah, this project, you know, I actually came into this project from a more computational background, but it basically involved me retraining as a wet lab protein biochemist. So I was basically completely useless for 3 months of my life. So I just wanted to thank some of my, the members of the Kim lab that helped me learn everything, and the co-authors on this paper are in bold here. And yeah, so, and thank you for listening. Happy to take any questions. I think there's questions in the chat, so maybe you can—

Thanks, Brian. Yeah, Lely is asking, have you tried this with AB-specific language models, example ESM fine-tuned on AB repertoire or the Sapiens model?

Yeah, that's a really good question. We're actually— we are trying that right now. I don't have the data, but yeah, we are looking at that.

Okay, great. To read the rest of the question, does using the general ESM model alone end up introducing mutations in the framework region that are typically germline constrained? And without having A/B repertoire germline knowledge, is it then more likely to introduce immunogenicity issues? I guess if you don't have the data, you can maybe— what are your thoughts?

Yeah. Well, we don't have any immunogenicity data. Put these antibodies in vivo or anything, or try to in silico predict them. Mutations in the framework region that are typically germline constrained. We actually looked at— yeah, so something I didn't mention actually, which is actually a very cool result from this, from an antibody engineering perspective, is that typically antibody evolution focuses on this part of the variable region called the CDRs that are directly, often directly involved in the binding interface. And there's another region of the variable region called the frameworks that are maybe less mutated during artificial and natural evolution. And, you know, half of the mutations that actually improved affinity were in these framework regions that are sometimes excluded from artificial evolution. So we look at like how often these framework regions are represented in— are in the germline. And it actually quite a bit are, and it's probably picking up that data from, you know, this— the small number of antibody sequences in the training corpus. But a few times, actually a number of times, Well, there's one big example where there's a mutation that's a glycine to a proline, and the glycine is seen in 99— like the abacus will say it's in 99% of sequences. And then, you know, 1% is the proline that was recommended by the language model, and that mutation was, you know, the highest fitness mutation for that evolutionary trajectory. I think it's both, yeah, to answer the question. Both germline constrained and potentially going beyond that. I can see the questions actually, so maybe I'll just go through them. So Mark's asking, can you elaborate on why the model works for proteins with different fitness criteria than antibodies? Yeah, so it's related to what people have been seeing in the literature already, that these likelihoods are just automatically— or I mean, they're correlated with fitness values that are specific in these mutational scanning datasets. I do think there's like some dimension of like an unreasonable effectiveness of unsupervised models in doing this kind of variant effect prediction. So as an example, there's this really cool work out of the Debra Mark's group where they use completely unsupervised models to do clinical variant risk and they train these models on variation of these proteins across different species. Fundamentally, there's not really— the definition of fitness in the clinical variant setting is very different from the definitions of fitness that would be encountered in, say, natural evolution. So I think there is something going on where there's some evolutionary prior that's really good for predicting certain definitions of extrinsic fitness. And so I think that's what these models are picking up on. Hopefully that kind of answered your question. Lynn Wong asks, is it possible to predict the outcome of VDJ recombination? I think that's a really cool project. I think these language models, Well, the language models we use would potentially not be the best solution to this. Like, it might actually— this might benefit from more traditional approaches, just, you know, based on multiple sequence alignment, maybe an antibody specialized to antibodies as well. But yeah, there is— we do know that, you know, VDJ recombination is not uniformly random and its recombination events. So, I think that's a really interesting question. Yeah, I think maybe beyond the scope of the models we use, but still maybe similar approaches would apply. So, Hannah is asking, it's really interesting that selecting 10 mutations, around 10 mutations, results in some fraction of hits, but I'm curious if you try testing as a negative control of sorts mutations that the language model predicts would be deleterious. That's actually a really good— yeah, so just like very low likelihood how many of those are actually true hits. We haven't tried that. Our experimental budget was actually, you know, quite constrained, so We're just operating in this very high likelihood regime, but yeah, it's an interesting negative control. Something that we have looked at is what hit rate would you get from random guessing, and then we do see substantial improvement over just potentially expected, but good to test, over random guessing. Joshua Elkington asks if there's a single script available on GitHub. There— yeah, the code— I still need to— it's a bit bare bones. There is a single script where you can edit the sequence in the script itself and then run it. So yeah, that is available, but something with like a nice API is something that I'm working on, definitely. And then Jay asks, do you think somatic hypermutation in germinal centers could serve as a biological analog for the greater affinity acquired by in silico antibody mutations, and would you expect that models would follow similar mutational trends? Yeah, I think— yeah, this is— we've actually thought about this question quite a bit, actually. And I think— so somatic hypermutation in germinal centers and, you know, AID-mediated mutagenesis— we know that even— this is a great paper from Fred Alt's lab where we know even in the absence of any selection, where selection can't be happening, there's very, very non-random mutation rates. And so potentially, you know, potentially those non-random mutation rates are what's being picked up on by the language models, like maybe some AID hotspots or something. And potentially the the immune system is picking up on or leveraging those non-random mutation rates to do something like we're doing by eliminating a lot of invalid sequences. But it's— yeah, looking at what actually happens in vivo and in germinal centers is something that I'm really interested in. Yeah. I guess you can also feel free to unmute if you have other questions. Otherwise, I'll just keep going down the questions in the chat. So Jason asks, it seems like you only did 2 rounds of evolution. Do you think that further rounds of evolution would continue to find improved variants? Yeah, since by predicting intrinsic fitness— since predicting by intrinsic fitness may only be somewhat correlated to extrinsic fitness, which seems like a limitation for I think that's especially since the language models are definitely only approximating like true intrinsic fitness. You might find yourself in a setting where if you're only acquiring from the totally unsupervised approach, you might end up in a, like a, like a local optimum or something. But if, I mean, I think a promising way to break out of that, that's been explored quite a bit in the community is adding some amount of supervision once you have, once you have that initial two-round of evolution training data. So, There's this paper, it's a great paper by Serge Biswas et al., the low-end paper, and I think they train on as little as 24 variants from error-prone PCR or something. So, maybe you could potentially start to supervise models from the— we require 20 variants, but maybe from that you can Maybe try to supervise to potentially break out of local optima and also try multiple combinations too. And then Nosos asks if you can simultaneously unmask 2+ positions from ESMM1D and get probabilities for the set of mutations. Yeah, you can. You can unmask multiple mutations and rank them that way, which is— which if you're looking to guide the combinations in our trajectory that way, you could definitely use that. You know, whether or not the— you'll get the conditional likelihood of the single site, but, you know, it won't— it's— I don't think it's very good at giving you like the joint likelihood of those two sites. It'll be like a product of two marginals, basically. So, whether or not that's a good prediction of the pairwise spatial effect might be a little more difficult, if that makes sense. But you could theoretically do that and try to approximate the joint likelihood by the product of the marginals, the conditional marginals. Oh, Brett asks, "As you pointed out, large functional improvements often require multiple mutations combined. Joe Thornton recently published a paper demonstrating that epistatic drift limits the predictability of variant combo effects. Can you comment how language models might extend to more than one mutation?" Oh, this is— I love that. The recent Science paper was a really great paper. So we've also discussed this quite a bit internally. Yeah, so fundamentally, well, I think that paper showed that if you have two different— it's the same protein family but two different genetic backgrounds, that as you diverge in the genetic background, if you diverge enough, then the mutational effects from like a deep mutational scan eventually become uncorrelated with each other. And so there's some amount of unpredictability if you try to condition on— this is my understanding of the paper— if you try to condition on one genetic background The other genetic background, which has drifted, would not be predictable from that initial one. I think the language models are doing something different because they require the genetic background to begin with, and then they predict the variant effects on top of that. So, it's a little bit of like a different— the definition of prediction is a little bit different in those cases. But yeah, that paper is really good at establishing like these strong upper bounds on— yeah, strong upper bounds on evolutionary predictability. And then extending to more than one mutation. Yeah, like it would probably, based on that paper, it would probably be easy to predict the effect of combos or triples or something. But if you're trying to predict the effect on fitness of large things, maybe it won't be good. I don't want to be like speculating too much, but yeah, it's very interesting to think of of evolutionary predictability from the light of these language models, because fundamentally they're epistatic models. They take a sequence context and they predict mutational effects from that. So there's something— yeah. Cool. Calvin asks, do the ML-evolved proteins also have improved proper properties in terms of aggregation and solubility besides TM? Actually, someone in the chem lab looked at improvements. So all of the, all the wild types that we saw were not— had no, just qualitatively, no, like, problems with solubility. But there's someone in the chem lab that was actually looking at a protein with known, like, issues with solubility and showed that actually, you know, some of the variants would— it would be the same thing actually that you would see with affinity. Some of the variants had a significant improvement in solubility, some had a significant negative effect on solubility. So it was like, that's something I maybe forgot to mention. So even if the mutations are having like a neutral effect on binding affinity or are staying within the range of like a therapeutically acceptable range of affinity, they might have some favorable property in another dimension that you could easily add to the assay. So even if they're, you know, not improving affinity, they could be improving solubility or TM or or a polyspecificity or something.

Great, everyone, let's thank Brian for the talk. And if you have more questions, we are going to hang out for a few extra minutes, so feel free to stay and ask more questions or just talk to the rest of us. Thank