Scientist Stories: Brian Hie, Learning to Read & Write Protein Evolution

Thank you all for coming as well. Yeah, unfortunately Bonnie couldn't be here, but you're stuck with me only for today, but hopefully it will be an interesting talk nonetheless. So yeah, today I'll be basically summarizing some of the work that I've been doing over the past couple years on really using modern evolutionary models, in particular protein language models, to both understand evolutionary rules, but then importantly also to then begin to translate those rules into better ways to evolve and design new proteins in the laboratory. All right, so let's get started. So as I'm sure all of you are probably aware, evolution can be a very powerful adversary. So for example, these new variants of SARS-CoV-2 cause these waves of COVID-19 and associated death. In a more extreme example, HIV can actually evolve its surface proteins to completely escape from the host immune system in a matter of days, actually. And of course, evolution is also a driving force behind the worldwide emergence of multidrug-resistant bacteria, as well as drug-resistant tumors. And so we go through all this work to develop these drugs and these vaccines for these different diseases, but, you know, in many cases, you know, taking COVID-19 as an example, evolution was actually able to render many of those drugs completely ineffective in a matter of months. And so if we want to develop more robust drugs and also even more robust vaccines, this will require certainly a better understanding of evolution, but also ways to translate that understanding into better engineering as well. So the path that I've been taking and the path that I'll be, I guess, describing in this talk is to learn evolutionary rules directly from data. And so for example, there's, you know, hundreds of millions of protein sequences that are publicly available. And these are sequences from proteins that have been undergoing natural evolution. And so one thing you might ask is, are there patterns in the sequence variation of these proteins that could reveal something about the rules and the evolutionary process that has produced these protein sequences? And the idea that you can learn evolutionary rules from data is actually a very old one. So if some of you are familiar with these amino acid substitution matrices, you know, BLOSUM62 as an example, the idea is that if you look at preferential substitutions between the amino acids and which ones prefer to be substituted for each other, you can recover things like, biochemical similarity between amino acids or distance in the genetic code is another thing you can recover. But these are very simple evolutionary models. Somewhat more complex evolutionary models are those based on sequence alignments. So this is the idea of you take some evolutionarily related sequences, you construct a multiple sequence alignment of these sequences, and then you look at sites that are more conserved, sites that are more variable. And from that, you can begin to discern some of the rules, for example, underlying the evolution of a specific protein family. And these models are actually very, are quite powerful. They're the underlying models underlying most of modern phylogenetics research, for example. But they do make this important assumption, this site-independent assumption, which can be limiting. And to illustrate this, this is one of my favorite illustrations to illustrate a limitation of this site-independent assumption. So I'm going to use a site-independent model to design a human name. And so if you look at the first name and you look at the most common first name across the global human population, for example, that name is Muhammad. So then, moving on to the middle name, the maximum likelihood middle name across the global human population is James. And then finally, the maximum likelihood last name across everyone on the planet is Wong. So Mohammed James Wong is certainly a very interesting name, but it's also a very rare one. And so what happened here, the idea is that the choice of the first name influences the choice of the middle name and the choice of the last name. And likewise, in biological sequences like protein sequences, you know, the choice of an amino acid at the beginning of a sequence influences the choice of amino acids in the middle and at the end of the sequence. And so this idea that there are these complex dependencies among the sites in a sequence, that sequence context is important, is referred to in the evolutionary jargon as epistasis. And so this brings us to the evolutionary model that I'll be considering and returning to multiple times during this talk, which is the idea of a protein language model. And so protein language models are trained with very simple objectives. So for example, there's a class of language models called mass language models, And these models take as input an amino acid sequence. And to train the models, you would randomly hide, say, 15% of the amino acids. So you would put a masked token instead of the actual amino acid and kind of hide it from the model in its input. And then the model is asked to predict the value of the masked tokens given the unmasked tokens in this input sequence. And then if you do this for thousands or even millions of protein sequences, you force this model to learn rules of how amino acids come together to form a protein sequence. And very importantly, you're using in a fairly direct way the information from different sites to make predictions about other sites. So you're directly modeling epistasis among among the sites in a sequence. And so in this talk, I'll be describing how these models can kind of learn evolution step by step, so to speak, so kind of building up slowly but surely. So in the first part of the talk, I'll describe how these models can begin to understand evolution, at least in terms of the smallest steps in sequence space, so single amino acid substitutions. In the second part of the talk, I'll summarize how these models could learn and understand evolution across larger jumps in sequence space. In the third part, I'll then begin to describe how these models can then guide artificial evolution in the lab, at least in terms of small steps in sequence space. And at the very end, I'll kind of look forward at potentially how these models could guide evolution over much longer-term evolutionary trajectories. But first, let's begin at the beginning and consider, you know, what about the very basic changes, you know, single amino acid substitutions? So to motivate this is the problem of viral escape. And so many viruses can actually readily evolve their surface proteins to escape from the host immune system, for example, by mutating the protein in a way such that a neutralizing antibody no longer binds. Viral escape is a very big problem. It drives the persistence of these incredibly deadly viruses, influenza, HIV, and now SARS-CoV-2, for example. And these viruses kill millions of people every year. Viral escape can also be caused by single amino acid changes. So for example, this histidine to serine mutation in this protein influenza hemagglutinin would render this broadly neutralizing antibody no longer able to bind, for example. And so when we were considering, you know, what a virus needs to do to escape, kind of like from first principles, there's a couple things you would want to consider. So if I'm a virus and I want to escape from the immune system, what do I want to do? Well, what I don't want to do is I don't want to pick changes that completely unfold the protein or otherwise make that protein unable to infect a host. But what I also don't want to do is I don't want to pick changes that make me look similar to the immune system so that, you know, for example, a neutralizing antibody can still recognize me and target me for neutralization. What I want to do, actually, is I want to maintain this balance between maintaining proper folding and infectivity, but I also want to alter the biochemistry of the protein surface, or I want to change somehow the structure of that protein so that a neutralizing antibody can no longer recognize me. And the key idea is that there's an analogy potentially in human natural language where you make mutations to, say, an English sentence where you want those mutations to still adhere to the overall rules or the grammar of that sentence. But importantly, you don't want those mutations to also be functionally synonymous. You want those mutations to, you know, change something about the overall statistics of the sentence and change the overall meaning or the semantics of that sentence. That's what we're calling semantic change. The idea that evolution can be modeled as a word game or has analogies to natural language is actually a very old one in classical evolutionary theory. For example, John Maynard Smith in 1970 introduced this concept of the word game and a very important a very important component or realization in this word game is the idea that there's this notion of validity or adhering to a grammar, what we're calling grammaticality, to evolution. So Maynard Smith's word game is, say you wanna mutate the word "word" to the word "gene" via single character substitutions., but importantly, all intermediates themselves need to be valid English words. And likewise, in protein evolution, all evolutionary intermediates also need to adhere to some notion of fitness or validity or what we're calling grammaticality. So Maynard Smith is mostly theorizing kind of from an armchair, But we can actually instantiate some of these theories with statistical models, and in this case, protein language models, and actually test some of these hypotheses out. So first, we hypothesize that the likelihoods learned by a language model are a good correlate of grammaticality. And that kind of makes sense. If you have high likelihood of a change appearing in some sequence context, that usually means that that change is grammatical or would adhere to the overall rules of that sequence context. But also, a language model learns an internal representation. So this is just a list of numbers or a vector of numbers. And also drawing inspiration from natural language processing, we kind of hypothesize that distance in the vector space or the embedding space would correspond to semantic similarity so that closeness in embedding space corresponds to semantic similarity and farness, I guess, high Euclidean distance between two of these vectors would correspond to semantic dissimilarity. So, importantly, a language model enables us to quantify both grammaticality and semantic change. So this brings us to the word game that we're considering in this to kind of model viral escape, which is what we're calling a constrained semantic change search or CSCS. And the idea is pretty simple. So you make all possible single token changes to a sequence of tokens and you use a language model to quantify both grammaticality and semantic change. And you just rank those based on high grammaticality and high semantic change and pick the top-ranked mutations. So this is a little bit abstract. So I have a couple examples.

Yeah, can I— OK, yeah. So I'll just wait until you've done the example before asking that question, actually.

I think that will— Oh, okay. Yeah, this is a little bit abstract. Oh, definitely feel free to interrupt me with questions. So this is a little bit abstract. There's a couple examples that we looked at that I'm providing just to make this a little bit more concrete. And this is from actually training a language model on human or on natural language news headlines. So for example, the original headline, Australian dead in Bali, The semantically closest single word change according to the language model is Aussie dead in Bali. So, you know, that is pretty much synonymous. The CSCS proposed change is Australian ballet in Bali. So you can imagine this being, you know, a valid news headline, but all of a sudden it's a lot more positive. Blastoff of Apollo 8 turns into blast victims of Apollo 8. And then, you know, wine growers revel in good season turns into wine growers rebel in flu season. So, this is a bit of a toy example, but it helped us kind of confirm some of our intuitions about how the statistical properties being learned by these language models correspond to notions of grammaticality or semantic change.

Yeah, so that's a good point for me to ask my question. So, in your objective, you have the ranking of the likelihood of the token given the rest. I guess when I think about grammaticality, I think of that as a binary thing, like either a word is a word, valid word, or it's like not a word. So I feel like the ranking by this objective somehow also encodes some of the semantic, like the meaning-y stuff. In addition to just whether or not it's valid. So yeah, it'd be great to hear your thoughts on that.

Yeah, I think, I mean, this is definitely a correlate of grammaticality, but it's not— yeah, I guess, and whether or not grammar is binary or whether it's continuous, I guess, is a big, also a philosophical question. Like Noam Chomsky might say that it's binary, for example, And yeah, and definitely if you're training language models on sequences of just regular text as it would be generated, for example, via normal kind of human text generation processes, this likelihood would also correspond to pragmatics as well. So sometimes words are used more often than others, even though they're both equally grammatical. So I agree that this is a correlate of grammaticality, but could potentially not just model syntax, but also pragmatics as well. So that's a good point. Great. So these are some examples for human natural language, and it's also a bit of a toy example. So now we're moving on to viral sequences. So instead of training language models on sequences of English words, we can also train them on sequences of amino acids. And so for this, we use, you know, the thousands of protein sequences available from viral surveillance data. And these are some of the results. So each dot here corresponds to a single amino acid change, in this case to influenza hemagglutinin. And it's plotted according to grammaticality on the x-axis as well as semantic change on the y-axis. And then we can compare these— we can compare the grammaticality and semantic change to experimental data. So this is deep mutational scanning data where, in this case, the Jesse Bloom Laboratory made all possible single amino acid changes to this same influenza hemagglutinin protein and then measured escape, in this case, from neutralizing human antibody sera. And so those experimentally determined escape mutations from this assay are plotted as the red Xs. As you can see, they cluster in this, you know, high grammaticality, high semantic change quadrant. And actually, the predictive power of grammaticality and semantic change is quite high despite the language model during training not having explicit access to what was an escape mutation, at least as defined in this assay. It instead learned these statistical properties of semantic change and grammaticality from the data that that we interpreted as such. And it turns out, you know, both were pretty predictive of escape. In the paper, we show that this generalizes, you know, to other viral proteins like HIV Env and potentially to SARS-CoV-2 spike as well. Something I like to do in giving these talks is, you know, highlighting the work of, you know, other groups that have, you know, built off of our publicly available code and models and, you know, even ideas. So for example, this team led by some authors at VEAR Biotechnology showed that, you know, actually language model likelihood or grammaticality was helpful in predicting mutations that spread in the global human population epidemiologically with, you know, pretty good accuracy even up to 4 months in advance. And then in this work led by, a team at BioNTech, who are the makers of a globally available COVID vaccine, showed that language model semantic change and log likelihood or grammaticality were helpful in developing an early warning system for vaccine development that could flag new variants of SARS-CoV-2 for the initial vaccine-related characterization sometimes months before those variants were designated as variants of concern by the World Health Organization. So, you know, always nice to see our ideas being useful in very practical settings. Another thing you can do when training language models is rather than training them on sequence variation within a family, you can train them, for example, on sequence variation across all observed protein evolution. And this is really enabled by industrial-scale compute. And when you do this, when you train these language models, say on the 90 million-plus non-redundant protein sequences from the UniProt database, for example, these language models could potentially capture evolutionary rules beyond a single protein family, but evolutionary rules across multiple protein families as well. So here is a plot where on the horizontal axis are different proteins. And for each of these proteins, someone has done a deep mutational scan. So someone has made many small changes to these proteins and measured their effect on some notions, and in these cases, actually quite diverse notions of protein fitness. So for example, BLA, this is beta-lactamase, that definition of fitness is antibiotic resistance, but there's other types of enzymatic activity or viral replication fitness represented here as well. On the vertical axis is the correlation between language model likelihood learned by a single language model and these diverse notions of fitness across many different proteins. And across many proteins, the correlation is actually pretty good in dark blue, and it's sometimes comparable in the orange to models that have been specifically trained on protein family-specific sequence variation. So this is telling us that potentially these models are learning something about evolutionary rules that are useful in predicting diverse notions of fitness according to these deep mutational scanning experiments. But then a question you might ask is, what about larger changes in sequence? Evolution doesn't just make single amino acid substitutions. It will make larger jumps in sequence space across longer spans of evolutionary time. So I guess I'll launch into the second part, but I guess—

I have a quick question about the previous—

Previous slide. Yeah.

So I guess it's interesting that the same model likelihood is predicting the mutations in all these different evolutionary contexts. Is the implication then that across evolution, despite the— it's dominated by the— these rules regardless of what is actually being selected for. Regardless of the function, it's the same pattern of mutations that you see.

That's only one very intriguing interpretation of this. I think in some sense the models can be are learning family-specific patterns and sequence variation. I think they're also learning more general rules of protein structure, for example, across all of protein evolution. And if you look at kind of the plots for most of these deep mutational scanning datasets, I think a lot of the performance in this correlation is actually the model can kind of tell the space of clearly unproductive mutations. And so I think that like separating functional things that maintain function from things that are clearly unproductive is driving a lot of this correlation. So I do think that if you just look at the percentage of mutations that are the theoretical percentage of the, in the theoretical space of mutations you can make to a protein, and you just look at the raw percentage of those that will be just clearly unproductive, most of them will be, you know, they'll blow up the protein, like cause it to unfold or misfold or something like that. So in that sense, yeah, I think a lot of the effort in evolution is in avoiding these, you know, clearly unproductive scenarios. Sequence regimes, and a language model could potentially be good at identifying those clearly unproductive regimes. But that's certainly a hypothesis at this point, I would say still.

Okay, yeah, so that's really cool. So it's like there's, you rule out the like the rubbish ones, and also you have like these proteins are in these deep mutational scanning experiments, they're kind of optimizing something that the protein already kind of does.

So you have that combination of it. Yeah, confounded in most of these deep mutational scanning experiments is the specific notion of fitness, surely, but then also whether the protein expresses and folds as well. So that the language model could be explaining that second part, probably, rather than learning the specific notion of antibiotic resistance to ampicillin or whatever.

Great, thanks. We have another question in the audience.

Hello.

Hey. Yeah, I just had a quick question. So I'm just wondering how to understand the usefulness of these scores or likelihood scores because you could consider two different mutually exclusive notions of fitness. For instance, you know, let's take 3ClPro in COVID-19. You could say fitness is what, you know, allows the virus to continue to propagate, or maybe fitness is what allows the enzyme to be— to have like the highest catalytic rate. And so you could have a mutation in the protein that decreases maybe its rate of catalysis, like its rate of enzymatic catalysis, yet increases like the virus's ability to to continue to propagate. So how does this score continue to be like well correlated with both of those types of fitnesses in a way that is interesting? Because one way you can imagine it's well correlated is just because it models n-grams really well, like the sequence conservation is there. And so like there's, you know, some probable, highly probable sequence, but like those likelihood scores, do they necessarily mean it's interesting, like if something has a high likelihood.

Thank you. I definitely think if something has high likelihood, it's probably interesting. I do, yeah, I do, yeah. If you wanted to say predict fitness or whatever for a protein with potentially many different definitions of fitness, then you would, have the same likelihood score, I guess, for both of those specific definitions of fitness. And potentially, if the fitness is very different from its natural fitness or something like that, or it's doing something weird, like maybe the definition of fitness requires some strange properties that go against prevailing notions of protein folding and physics or something like that, like the correlation won't be very high. But most definitions of fitness rely on things that are kind of general or at least properties of the protein family itself as observed in past evolution. And those properties tell us something about potentially very diverse notions of fitness from what's been happening in nature. So I think even if the— definition of fitness is very different from what's seen in nature, these likelihood scores could be helpful. And there has been work from the groups of, you know, Deborah Marks and also Vasilis Ntranos showing that, you know, if you just train evolutionary models on sequence variation from diverse species, so, you know, various prokaryotes and single-celled organisms, and those likelihoods are helpful in predicting clinical variant risk. I think that those definitions of fitness, like clinical variant risk, might be very different from what's going on in these natural prokaryotes. So I don't know. I think it's an open question. I think there's a lot of uncertainty. And I'll get to some of this later on in the talk, where where I kind of describe the utility of likelihood scores in actually guiding evolution. But it's definitely a good question and something to consider.

Sorry, I'm just going to— no, that was a really good answer. Thank you. I just wanted to follow up with something. So I know some machine learning groups in the past have shown that models, for instance, trained on CIFAR assign high probabilities to MNIST or something.

Oh, oh, oh.

How do you think that would manifest in the scores that you're assigning?

And would it be beneficial or negative? Yeah, and I think you brought up like n-grams and stuff. I think that is also probably something definitely that the language models are picking up on, like common motifs and stuff like that that have been observed in past evolution. And I can see those being very helpful in kind of learning, you know, general, you know, what these rules are potentially could be explained by n-gram models, but it's not the only thing that these language models are learning. They can go beyond trivial combination, like trivial learning of n-grams, for example, but they're definitely doing that as well, I think. Sounds good. Great. So I guess moving on to the second part of the talk, we're kind of building up step by step. So that was the And the first, that was the first part. I guess we did get bogged down in questions a little, so I might need to go through the talk a little at a faster clip. But here what we're considering, you know, what, how these models can understand evolution over a much longer-term evolutionary trajectory. So for this, again, we can go back to notions from classical evolutionary theory. So these are ideas of the fitness landscape first proposed by Sewall Wright in the 1930s. But, and the key idea, I guess, underlying these notions of a fitness landscape is that at each point in sequence space, a sequence will be trying to make a locally optimal improvement in fitness. And so, if you could potentially have a local predictor of evolution, then if you compose these predictions across a larger sequence space, then you could potentially discern something about more global patterns across this sequence space. So this is our idea. Say you have two sequences, sequence A and sequence B. And a language model might say that the mutations observed in sequence B have higher likelihood of appearing in the context or the background of sequence A than vice versa. So importantly, this difference in language model likelihood allows you to assign a directionality between two sequences. And then across a sequence similarity network, so this is where each node corresponds to a sequence and it's connected to, say, its k nearest neighbors in sequence space. You can assign a directionality to each edge in this network and then zooming out over the entire network, you can analyze this network for patterns and the overall flow of evolution through the network. And so this is an approach that we refer to as evolutionary velocity, kind of inspired by ideas of RNA velocity in single-cell transcriptomics, if some of you are familiar with that. So for one example, this is influenza nuclear protein as a starting case. So nuclear protein is very interesting because it evolves at around a single amino acid change a year, and its evolutionary history is very densely sampled via viral surveillance. So we actually get pretty good and complete evolutionary history of this protein going back decades. So here is a sequence similarity network of this protein. Each node corresponds to a nucleoprotein sequence. It's colored by its known sampling time. And then overlaid on top of this visualization is the flow of evolution as predicted by evolutionary velocity. So at least in terms of the visualization, in many cases it corresponds to the known temporal evolution of this protein. But we don't have to be limited to visualization. We can quantify this. So here are the root nodes of the directed graph. You can kind of determine them via a network diffusion process. These are basically the nodes that the direction of evolution is constantly flowing away from, so to speak. And so for example, these root nodes actually correspond to the known zoonotic crossover events in influenza history. So for example, we have 1918 pandemic influenza, 2009 swine flu, and 1960s H3N2 as well, as well as H5, H7 trying to break through in the 2010s, but hopefully not yet. And then we can compute this notion of pseudotime. This is basically distance from a root node. It's a way to order the sequences based on distance from the closest root node. And if you, correlate this notion of pseudotime with known sampling time, the correlation is pretty good despite the model or the approach and the model not having any access to known sampling time. It instead inferred these statistical properties from the language model likelihood throughout the network, and it turns out that it was able to reconstruct known sampling time pretty well despite not having access to it. Pretty surprising. Pretty surprising.

What's the training data for the, for the inference model?

Yeah, so the model is trained on UniRef50. So it's the UniProt database downsampled to 50% sequence similarity. Actually, if you look at nucleoprotein, there's one example of it in the training dataset. So it's actually the example from 1918. And we do show in the paper actually that sequence similarity to training dataset does not explain, I guess, the direction of evolution in these, as predicted by this model. But that is a good point to make. Definitely consider potential bias from the training data. Is there a second question?

Is it on?

Yeah.

Hey, on the left side, or the left part there. So I wonder, in my head, the evolutionary pressure that guarantees for this shared directionality of mutation is sort of like a somewhat shared trained immunity environment, so maybe like somewhat shared exposure history of a population. So I wonder how you think about that conserving over generations, right? Because influenza, you talk about like 100 years. So how do you think about directionality being conserved over that timescale?

Yeah, that's very interesting. And there's potentially many different selection pressures operating on this protein. So, you know, immunity is definitely one selection pressure. And, you know, other selection pressures are more basic, like proper folding and stuff like that. What's nice, what's kind of nice about these models is that they don't assume specific selection pressure. And potentially they can capture many of these selection pressures and summarize them in a single likelihood score. So I think the empirical result we're seeing is that the direction of evolution predicted by these language models locally and then assembled across an entire network are empirically helpful in predicting directions of evolution and natural evolutionary trajectories. Speculating as to how this corresponds to specific notions of selection, I don't kind of want to get into that because it goes beyond the experimental data that we have. But it's definitely interesting to consider how this direction relates to consistency of selection and things like that. Yeah. Great. So yeah, and related to that question, I guess, is somewhat surprisingly, this approach seems to be able to predict the direction of evolution even across much longer evolutionary trajectories. So for example, cytochrome c is a highly conserved protein across most eukaryotes. Actually predicting pseudotime could on average, I guess, reconstruct the order of these different taxonomic categories in the fossil record. And this approach also enables us to weigh in on evolutionary controversies, I guess, or evolutionary uncertainty. And so for example, serpents are a highly conserved protein family, And unlike many highly conserved proteins, in the case of serpents, there's actually much more eukaryotic diversity than prokaryotic diversity. Usually it's the other way around. And so this has led some to hypothesize that maybe in the case of serpents, the prokaryotes actually acquired serpents from eukaryotes via horizontal gene transfer, rather than the traditional notions of prokaryote to eukaryote inheritance. But actually when we run this method on the landscape of serpents, it indeed also places the predicted root in the prokaryotes more consistent with a canonical evolutionary history for this protein family. And then the paper, we also apply this to other interesting evolutionary cases, and again highlighting some of the work that is built off of our publicly available code and models, these groups, this is Nozomi Ando's group from Cornell and Colin Jackson's group from ANU, also use evolutionary velocity alongside traditional phylogenetic methods to argue for an ancestral class of ribonucleotide reductases, kind of this interesting enzyme family. So I guess I can again pause here for questions. Yeah, I didn't mean to discourage questions. Definitely keep the questions coming. If we have to cut this short for time, that's probably fine. But definitely ask questions or keep them in mind. But if there are no questions, I can— Well, I have a quick question about your UMAP plot. Yeah. Could it be the case that they are sort of circulating, like there is no root really, like it's just circular? Yeah, that's interesting. I guess— In theory, your model does not guarantee that there will be a root or even just multiple roots, right? That could be interesting. I guess because evolution is a temporal process, you could, in theory, has a root, but you're right in that sometimes there could be no root or there could be lots of convergent evolution. So, and actually something somewhat rather suggestively, the direction of evolution for H5 and H7 actually points inwards and it goes closer to 1918 influenza nuclear protein in this UMAP plot. And it predicts a different route here, and then the direction of evolution kind of going towards 1918, potentially indicating convergent evolution. And actually, in theory, this could be a good— this could be modeled by this kind of analysis. If you have, you know, two independent evolutionary roots or something, and there's actually convergence, something like this could possibly detect that. Whereas if you're doing traditional phylogenetic analysis, that assumes a single root to the tree. And so there you would not be able to explicitly model, or it would be very difficult to model convergent evolution. So it would be interesting to see like the equivalent of a circular cell cycle or something in evolution. I think that would be a very interesting, that'd be a very interesting finding. Or if influenza nuclear protein eventually circled back based on population level immunity waning to some ancestral protein, that would be very cool to see. And it could be captured by this approach in a way that traditional phylogenetics couldn't. So that's an interesting one.

Point.

Thanks.

Great.

So I'm going to start to pivot at this point from understanding evolution as it's happened. Oh, question. Thanks. That velocity field picture reminded me of a paper that looked at cell differentiation and the vector field that represents the differentiation and pseudotime aspects there. Do you think a lot of your methods might have— or some of your methods might have applications to cell fate or similar kinds of problems? I think, yeah. In this case, we were inspired by methods for determining cell fate and things like that. So, I mean, underlying both is the idea that a prediction of local directionality can give you a picture into potentially more global directionality across a network. So, in RNA velocity, for example, you're making local predictions based on splicing kinetics or something like that. Here, we're making local predictions based on language model likelihood. But I think the idea that you can get some insight into a global picture of dynamics via local predictions is what's unifying these two ideas. So it's really interesting when you kind of think about it that way. Yeah. OK, very cool. Thanks. I finally— I realized I didn't have access to the Broad or wasn't displaying the Broad video feed, so I couldn't look at— people asking questions, but now I can see it. All right, if there's no more questions, I'll begin to move on to guiding artificial evolution in the lab. So one question that we had when seeing that these models can predict the direction of evolution, at least retrospectively, is could they also be useful in guiding artificial evolution of new proteins in the lab. So you can do a very simple experiment. So the simple experiment is what would happen if you picked amino acid changes with positive evolutionary velocity, so to speak, or these are just mutations with higher language model likelihood than wild type. So our main experimental test case for this is the artificial evolution of antibodies. And so in vivo, actually, the immune system will evolve an antibody to have higher affinity against foreign antigen. And it's a pretty cool in vivo evolutionary process in germinal centers. And likewise, inspired by this, as antibodies are becoming a more attractive therapeutic modality people are also interested in artificially evolving these antibodies against therapeutic targets. And these methods typically use very high-throughput techniques like phage display or cell surface display to test thousands or even millions of antibody variants for the antigen of interest. So something that we were interested in, especially in terms of more efficient evolution, is what would happen if, if we use a language model to guide the evolution of antibodies, and that could potentially allow us to test instead of thousands or even millions of variants, around 10 or so. So I'll walk you through the results, and this is what we did. So for example, this antibody, Medi8852, is a very highly engineered antibody that binds to the hemagglutinin protein in influenza virus. So in this round, each light blue dot corresponds to a variant of MEDI8852 that's a single amino acid substitution from wild type. And the wild type is this gray dot here. So when we tested 13 of those variants in the lab for binding to influenza human glutinin, and each of those mutations was recommended by a language model, I might add. 2 of them actually improved affinity against influenza hemagglutinin. And this was pretty surprising because the language models have no idea that these sequences bind to anything, you know, let alone this specific strain of influenza hemagglutinin. So then in the second round, we combined the single amino acid substitutions from the first round into a single variant and show that this also has an improvement in affinity. We're using a technique called biolayer interferometry. So these are the traces from BLI. The gray corresponds to the wild type. The dark blue corresponds to the best machine learning guided variant. And screening across a panel of diverse hemagglutinins from different viral strains, we observe up to a 7-fold improvement in affinity despite this model not having any explicit knowledge of the antigen. MEDI8852 is actually a pretty challenging starting variant because it's so highly evolved. When we looked at the unmutated common ancestor of MEDI8852, so this is the germline kind of much lower affinity starting point of MEDI, and testing 8 variants in the lab, 3, maybe 4 of those mutations also improved affinity in terms of single amino acid changes. So in the second round, we're selecting for mutations that improve affinity. And so each dot corresponds to— or the dots correspond to all possible combinations of these affinity-enhancing variants. And we observe additive. Even synergistic sometimes effects on affinity in the second round. Again, screening across a diverse panel of hemagglutinins, we observe up to, you know, a 23-fold improvement in affinity for one of those hemagglutinins. We also tested for generality, so we screened diverse antibodies against diverse viral antigens, also the glycoprotein of Ebola virus, and and the spike protein in SARS-CoV-2. In all cases, we observed modest to substantial improvements in affinity. In the highest case, this, you know, 2 orders of magnitude improvement for this Ebola virus antibody. But importantly, this is doing very, very efficient or, you know, lightweight directed evolution experiments. So for each antibody, we're only measuring 20 or fewer new variants across just 2 rounds of evolution. And so this is in contrast to the typical paradigm of measuring thousands or millions of variants via the high-throughput techniques. We can also show that these improvements in affinity are not just explained by trivial polyspecific binding to proteins in general, So this is a polyspecificity assay described in these two references. You can test basically polyspecific binding to soluble membrane proteins via flow cytometry, and we observe no significant changes. And then in this interesting preliminary work led by a student in the Alice Ting lab, we can also show that the same model that was helpful in improving the affinity of antibodies could actually also improve the catalytic activity of an enzyme. So in this limited screen of around a little over 20 variants, 2 of them actually improve catalytic activity, in one case by as much as 10-fold. And so actually, the language models we're using to guide the artificial evolution of antibodies are not specialized to antibodies. So they're also trained on generic protein sequence variation. So again, potentially they could be helpful in modeling different notions of fitness as well. And so, you know, actually for this, I just swapped out the antibody sequence with the sequence of this enzyme, and it turns out that in this limited screen, that two of those variants also improved enzyme activity as well. So I think this is pretty interesting preliminary signal And it really helped the student out. I think he had done, you know, yeast display, an error-prone PCR and gotten nothing that improves activity. But finally, on this limited screen with a better assay, a low-throughput assay, he was able to improve activity. All right. So again, I can pause for questions, but I might be out of time, at least the scheduled time. So maybe I can blaze through this last part, or if we are fine going over time, that's okay.

Yeah, let's go through the last part and then we can save questions for the discussion, which I encourage everybody to stay for.

Great. And this part is also a little bit more forward-looking as well. And so, but I think it's also, it has some interesting ideas potentially. So this last part is, you know, we've seen evolution guiding, we've seen how these models can guide at least small changes in sequence space, but what about long-term evolutionary trajectories? Inspired by, you know, long-term thought experiments like Replaying the Tape of Life by Gould, I won't get into this too much, and also by these long-term evolutionary experiments in vitro in the laboratory, for example, Richard Lenski's long-term evolution experiment in E. coli. But these experiments take 35 years to run. Potentially, we would want a much faster way to do evolution, especially for designing things in the lab. One potential way of doing this is moving a lot of evolution in silico. So for example, there's these very simple algorithms inspired by evolution, like genetic algorithms, where you say initialize a random sequence, for example, you loop for some number of generations, you mutate that sequence, you compute an in silico fitness function, f, and then if that fitness is higher than the best current variant, you set it to the mutated variant and then continue. There's variants of this like Markov chain, Monte Carlo, where say you tolerate some decreases in fitness to kind of break out of local optima. This is a very, very simple optimization algorithm. It's just black box optimization via MCMC. So if we're going to have any hope of this algorithm actually designing things that are interesting and biologically relevant, we will need a very powerful fitness function, f. And importantly, this fitness function should be a very high fidelity replica a fitness as observed in nature, or at very least, fitness observed in a lab, and hopefully, we'll also be able to report its uncertainty. So as a very basic proof of concept, and this is still work in progress, one fitness function that is becoming much more powerful is the prediction of protein structure. So for example, the AlphaFold neural network can predict atomic-level protein structure, in many cases, approaching experimental accuracy or at experimental accuracy. It turns out that this first part of AlphaFold relies on evolutionary information or greatly benefits from evolutionary information from multiple sequence alignment, for example. Language models also learn evolutionary information. So one thing you can do is replace that evolutionary information with the MSA in terms of the MSA search with a language model. And you still get pretty accurate atomic-level structure prediction, lower than that of AlphaFold, but still pretty good. But importantly, you get much faster structure prediction. So you can do things like predict 600 million protein structures in 2 weeks, for example. But another thing you can do is put efficient atomic-level structure prediction into your in silico fitness function. For example. And so we do this as a very basic proof of concept. What if we can evolve in silico interesting de novo structures? So this is an example of an in silico black box optimization of a structure where we define fitness as actually a rotational symmetry in the structure. So in this case, 5-fold rotational symmetry. And each image in this GIF corresponds to a different generation in that MCMC kind of in silico evolutionary process. At the beginning of the optimization, this protein is not symmetric and it's colored red, indicating low model confidence in the predictions. As the optimization progresses, it's green, corresponding to high model confidence in the prediction. As well as it becomes much more symmetric and adopts this actually cool star-shaped pattern. The idea of guiding in silico evolution via fitness function is actually very general, and it's very easy to specify that fitness function. So we can do things like very easily vary the level of rotational symmetry that gets designed from 3 to 8-fold, for example. Going back to different, those ideas of replaying the tape of life, each dot here corresponds to a different replay, so to speak, of the tape of life. Here on the vertical axis is TM score to the most structurally similar structure, experimental structure in the PDB. A TM score of 1 corresponds to perfect structural alignment and TM scores below 0.6 and especially below 0.5 correspond to substantial difference in structure from anything that's been experimentally characterized. So we do find generations from this approach that resemble things observed in nature, but we also observe things that begin to go beyond anything that at least has been experimentally characterized in the PDB, which is pretty interesting. And then finally, the idea of guiding in silico evolution via this generic fitness function through black box optimization allows you to actually compose potentially many constraints into that fitness function beyond just symmetry, for example. So you can have symmetry, but then you can constrain atomic-level coordinates to match the sites of something from nature, or you can design asymmetric complexes consisting of 4 chains where 2 of those chains are symmetric to each other with 2-fold symmetry and the other pair of chains as well, but then they're asymmetric to each other. So quickly these constraints kind of become unmanageable. And so we also propose this high-level programming language for protein design that basically enables you to have some kind of sanity in your organization of the constraints and kind of reason about these constraints in a hierarchical or somewhat more controlled and, you know, specified, hierarchically specified setting. I want to just briefly conclude. I know I'm over time, but the idea of simulating evolution via structure prediction is what's demonstrated in this proof of concept. But as function prediction also gets better, you can imagine doing this with efficient and powerful function prediction as well. So for example, as we get better prediction of protein-protein interactions, for example, you could potentially model these evolutionary trajectories, actually coevolutionary trajectories of antibody-antigen coevolution, potentially simulating coevolutionary trajectories that we actually observe in HIV patients, for example. Great, so in summary, I've described how these models can understand evolution in terms of small changes in sequence space, large changes in sequence space, and also how they can guide evolution in terms of small changes in sequence space and potentially guiding evolution over much longer-term evolutionary trajectories as well. I glossed over a lot of information for the sake of time, but we have publications or preprints describing these efforts, and I just direct you to these references if you're interested. Great, so that concludes the talk. I want to very quickly acknowledge a very talented group of co-authors and collaborators that span many institutions. And their names and some of their faces are provided here. And thank you all for your attention. Looking forward to discussing this more in the follow-up discussion.