Scientist Stories: Chloe Hsu, Deep Learning in Structural Biology & Protein Design

Thanks everyone for coming to the final tutorial session of the week. Today we have Chloe Xu from Berkeley. Chloe is a PhD student here advised by Jennifer Lisgarten and Moritz Hart, and Chloe has done a lot of work on applying deep learning methods to structural biology, and she's going to give us a tutorial on this. Since Chloe is presenting over Zoom, I will be moderating questions here, so I'll try to keep a lookout for audience questions. And you can tell your question to me and I'll repeat it over the mic. And if this doesn't work, then we will take questions after the talk.

Thank you.

All right, thank you, Pritam. Yeah, hi everyone, my name is Chloe. I'm really sorry that I'm in Berkeley right now but tested positive for COVID, so have to do a remote talk. Can everyone hear my volume okay? Yes. Okay, great. Yeah, and during this talk, please feel free to ask questions, and I hope the remote setup works out okay for questions. So maybe before I jump into the talk, a little bit about my background. I started my research originally in theory and complexity theory. So it, it makes me very happy to see everyone at Simons, and it has a special place in my heart, even though I've moved on to biology. I, during my PhD, I switched from machine learning theory to start working on computational biology, especially using deep learning methods for structural biology and protein design. So today I hope to give maybe a mix of both. I plan to cover the recent progress and what's going on right now in the world of using machine learning for biology, but try to maybe also focus more on the underlying assumptions or the more mathematical parts of it. So the focus of today will be on proteins and structural biology, but before we get more focused on structural biology or proteins, I wanted to first give a broader view of why we care about these problems and perhaps a higher-level picture of what are the interesting, important questions in biology, and why do we care about those. So if we look at ourselves, we're all humans, and there's a lot going on to keep us alive and functional. There are all these different systems responsible for getting air, for fighting viruses or foreign invaders, and then for digestion, for, for pumping out toxins through our kidney and then through our system. So there's really a lot going on. And while it's easy in studying proteins or structural biology to focus on particular molecules, the message here is that we are really complex systems and there's a lot going on. And it's perhaps today not quite possible to model all of the complexities going on in human biology through computational models. So we have to do some abstractions and then focus on the parts that would be meaningful to model and we can actually model today. There are all these systems, and then if we zoom in a little bit from the global systems in human biology, then in every part of the system there's still a lot going on. For example, the picture shown here is one region in the small intestine, so that's one small part of the entire digestive system. But if we zoom in, then there are all these different layers on the small intestine, and then if we further zoom in to to the wall of the small intestine, then we see these further finer structures on the surface of the intestine. Okay, so now we're zooming from the whole body to a very particular part of it, but then there's still a lot. And then if we take one particular cell on the intestinal wall here and zoom in, then we see there's a whole nother world in the cell, and there are all these different parts in the cell responsible for keeping the cell alive and functional. And then if we further take one particular part of the cell and zoom in, we see, okay, there are actually so many molecules working together even in a particular subpart of the cell. And now with structural biology, When people say structural biology, usually means studying protein structures, protein-protein interactions, and the 3D structures of particular molecules. So in structural biology, we're really looking at only one very small thing at a time. We would be looking at, let's say, this particular yellow thing here or this particular purple small molecule dot over here. So I think it's really humbling to keep that in mind when thinking thinking about structural biology that while structural biology has made a lot, a lot of progress, and it's amazing that the things that we're able to see now, but we're still very, very far from seeing the whole picture of what's going on in human biology. And it'll probably take a lot more breakthroughs to be able to have a fuller understanding of how things actually all fit together. So I'd say there has been a lot of exciting recent progress, but we're definitely still in the early days in the grand scheme scheme of things. So now I guess what we're able to do is that we can have a much better understanding of each of these small parts in this really colorful and complex picture, but we're still pretty far from understanding how these different parts work together, or even where each part is in the cell. This is not a real picture taken under a microscope, but rather an artistic rendering. It's still not possible today to actually look at the cells through imaging at the spine level and then see which part goes well. We just have an overall idea of there are these thousands of different proteins that work together in a cell, but we don't actually know where they are and what they're exactly— where they are and how they exactly work with each other. So that's kind of the overall picture I wanted to give as a background. If you think back to high school biology, you've probably heard about central dogma where DNA transcribes into RNA and then gets translated to protein. So I think this is still very much true, the DNA, RNA, protein part, but we have in recent years come to understand that there are actually a lot more to this. It's not just the DNA, the genetic code, that determines how we are and how we function. There are actually many chemical modifications on top of DNA bases or RNA bases or on top of proteins. So the, the perhaps, at least when I was growing up in middle school, high school, we were taught that there are ATCG, 4 bases, and then it's similar to instead of a binary code of 01, you have these 4 letters as the alphabet in a string and that determines life. But over the past couple of decades, the research community as a whole have made a lot of discoveries outside of this perhaps traditional representation of biology and actually a lot of other chemical modifications or changes also affect things at a very fundamental level. So while given all these caveats, it is still very much true that proteins are the main workhorse of biology. There are 20,000 proteins in humans. If you think about it, 20,000 is perhaps Not a small number, but not a huge number. Our whole body, there are these 20,000 different proteins in different parts doing everything. To me, that seems pretty amazing that we can cover all these diverse functions and we're all here and alive with just 20,000 different basic units of proteins doing all the different tasks. Yeah, it, to me, it's really amazing that at the basic level there are these different proteins doing an incredibly diverse set of tasks to keep us all alive and here and understanding how that works and even beyond these naturally existing proteins, designing new proteins to do other tasks for therapeutic or environmental applications. That's really exciting. Okay, so that's my introduction. Introduction before getting to the machine learning, maybe I'll pause here to see if there are any questions. Okay, then I'll continue. I'm guessing a lot of you might have heard in the news about AlphaFold making a lot of progress on the CASP protein folding structure prediction benchmark. I'll cover some of that today, but we'll also talk about some other recent advances in using machine learning for structural biology. If we look at protein structures, there At the basic level, made up of amino acids, there are around 20 different amino acids that make up the building blocks of amino acids. If you go to CVS or Target and then you see a bottle that says glutamine, for example, on it, that's one type of amino acid. Now, these Linear chains of amino acids then get shaped into what we call secondary protein structures. They could either be these sheets, beta sheets on the left, or they could form a helical structure called alpha helices. And then these secondary structures then further fold together into tertiary protein structures where the The sheets and the helices then twist and turn to form these structures. So when we hear about protein structure prediction, or let's say AlphaFold, what it's doing is to go from this first level of linear chains of amino acids to the tertiary protein structure. But it actually doesn't end there. To perform most important functions in humans or in life in general, the tertiary structure is not enough. Proteins then aggregate into what we call quaternary protein structure, where there are more than one chains of amino acids forming together into large assemblies to perform protein functions. On the left, for example, it's a pretty crazy protein assembly where there may be a hundred or dozens of protein chains formed all together into this pretty big ring. This is from the Salmonella bacteria, the same Salmonella that causes stomach issues. And with this entire big protein structure, it actually is able to allow the bacteria to swim or to move. This whole thing is a motor for the bacteria that allows it to move. So as you can see, there are so many chains together, and it's also not a static structure. It's a pretty dynamic structure that can actually move, and then the outside of the ring allows it to function like a motor and allows the bacteria to move. So we have made a lot of computational advances in predicting the tertiary structure, and I would say now since AlphaFold 2, the recent focus in the past couple of years have been moving above tertiary structure to also use machine learning and other computational methods to predict the protein assemblies, the higher-level structure of how different protein chains interact with each other, and also the dynamics of how these protein chains move and perform dynamic tasks. No, there are a lot of proteins. They're all made up of the same amino acid building blocks, but they can be folded into very, very different shapes. And just to give an idea of how different the shapes could be, there are some examples here. For example, the tube on the right that's made up of proteins, and it could form into this assembly of a very long tube by stacking together the protein building blocks one by one. Or there are some string-like formations where it's stacked one over the other, or it could be aggregated into a big ball with a lot of different protein chains on the surface. These protein structures over here, they're gathered over the past 20, 30 years in this data bank called the Protein Data Bank. And without Protein Data Bank, we wouldn't be able to do a lot of the computational or machine learning advances on protein structure prediction. I think it's often very easy to look at the whole dataset and then see, okay, we have 100,000 different protein structures in a databank and each one of them is just a data point. But if you think about the history of how these protein structures are generated, usually back in the day, if you resolve one new protein structure, that could be a Nature or Science paper. The Protein Data Bank of, say, 100,000 different proteins, that's really a lot of work by a lot of scientists resolving them one by one by hand, and there's thousands or maybe tens of thousands of papers behind this entire data. It's a really remarkable achievement of the entire community to generate all this structural data. Also put it together in an open source and standardized format to allow the whole community to use it and also do machine learning on it. Great question.

Yeah, the question—

so formally, what does predicting the structure mean?

Does it mean like predicting the 3D locations of the individual amino acids? That's what the question, to repeat, is. Formally, what does it mean to predict the structure? Does it mean to predict the 3D locations of all of the acids or something else?

Yeah, that's right. So the input of the problem would be a linear string of the amino acids, and then the output would be 3D coordinates for every amino acid. So a finer level of detail here is that within each amino acid, there's— there are usually 10 or 20 different atoms. So perhaps the more detailed version of protein structure prediction is input the string of amino acids and then output 20 different coordinates, or however many atoms there are. Let's say if there are 20 atoms in every amino acid, then 20 different 3D coordinates for each, each of the atoms in each amino acid. So This has also progressed over the years. Earlier, usually people take some abstraction and then represent each amino acid only as one dot or a few atoms. But one of the reasons why AlphaFold was so impressive is that it's now able to have a finer level prediction for all the atoms. And not just the main atoms in every amino acid.

Another question. I have sort of a related question. It's like, what does it mean to resolve one of these by hand? Is there like a certain, like, what does that look like mathematically? Do you have to like solve some giant system of equations or something like that in order to figure out how these structures fit together?

Yeah, that's a really good question. So yeah, it's not possible, at least in my understanding, it's not possible to do an experiment where we can directly measure the 3D coordinates of every atom one by one. But rather, the, the two main technologies for measuring the structure is by X-ray crystallography or from cryo-EM. And in both cases, we would do, let's say, X-ray crystallography, there would be a bunch of outputs from the experiments. But then the outputs of the experiments need to be translated into the, the 3D coordinates of the protein based on some equations or some underlying assumption of how the 3D structure translates to the, the direct measurements in X-ray crystallography. And it's usually not a very trivial process. There are there are a lot of software packages designed to do this, but it's a pretty involved process to interpret the X-ray crystallography or cryo-EM data. And I'll also touch on this later. One of the most immediately exciting uses of AlphaFold is actually to help solve X-ray crystallography or cryo-EM data, because in some cases when it's a very complicated protein structure, for example, one over here when there are a lot of chains and a lot of things interacting together, then it might be the case that we are able to do the cryo-EM or X-ray measurement, but we don't really know how to interpret the data or find a solution of the 3D structure that would satisfy the experimental outputs. So one of the use cases for AlphaFold right now that a lot of people are using it for is to generate predictions for the structure and then use that as a basis to help solve the experimental measurements.

Another question.

I was just curious about the data makeup of the— is it like, yeah, the ratio of synthetic to naturally occurring proteins in the databank?

It's mostly naturally occurring proteins. And actually, that's a really good question. So one of the problems with the protein data Protein Data Bank is that it's heavily biased towards human proteins and towards certain model organisms where people study a lot. There are some synthetic proteins in Protein Data Bank, but it's a very small fraction, and not all proteins can very easily be crystallized. So there is also a bias in terms of what kinds of proteins get put into Protein Data Bank. So for For example, some of the proteins that are more flexible and move around a lot to perform its functions, those are harder to crystallize. And so there are all these data biases in protein data bank, but it's pretty interesting that so far it seems like those biases don't seem to affect the performance the generalization of the machine learning methods too much. Definitely there are some influences from the biases. Maybe say AlphaFold does perform better on human proteins because it's seen more training data, but so far it seems to, at least to me, it seems surprising how well it generalizes to other things given the bias in the training data.

Oh yeah, another question.

Yeah, how rigid or flexible is each amino acid? Like you were talking about identifying the location of each atom in an amino acid, and I guess my naive guess would be maybe that once you've identified a few locations, just the rigidity of the amino acid might specify the remaining ones. But am I incorrect about how rigid these things are?

Yeah, that's also a good question. Thank you. It is rigid to some extent. So let's say even though there are 20 atoms, it won't have a full 20 degrees of freedom, but maybe it'll have— I'm not really sure about the exact number— let's say 6 degrees of freedom. So, um, it— yeah, there, there are definitely some, uh, there definitely is some rigidity to help the prediction. And in AlphaFold, even when it predicts all the atom positions, it also makes use of these rigid transformations to help learn the position of all the atoms. But the rigidity is not perfect. So even if, let's say, let's reduce the 20 degrees of freedom to 6 degrees of freedom, it doesn't— there is still some relatively minor shifts in the position that could happen. So it's not fully rigid.

Any other questions? Go ahead, Chloe.

Cool. Yeah, these are all really great questions. Thank you all. Okay. So yeah, yeah. As— yeah, that was a great question for segue into this. So yeah, I think One, maybe this was not as talked about in the media or in the public discussion, but for people in structural biology, I think the most impactful use cases of AlphaFold so far is to interpret complex experimental data, especially for cryo-EM, where cryo-EM is a relatively newer technology for doing structure measurement compared to X-ray. X-ray crystallography, but it has a lot of advantages. For example, it allows people to look at more dynamic pictures of proteins, and that helps us understand the dynamics and how functions are performed with the dynamics. But it could— I guess before AlphaFold came around, for a lot of the more complex cryo-EM measurements, it could take people months to try to work out the math and solve the cryo-EM measurements. But right now, AlphaFold is still not perfect at this, but at least it could be used as part of the pipeline to help speed up that data interpretation process by a lot. Another use of structure prediction is to generate a lot of new data for predicted structures. And actually just last week, DeepMind released 200 million predicted structures from AlphaFold for all kinds of organisms. So that covers a much wider basis of the structure universe, not just focused on human or E. coli structures, but it covers more organisms and give us a fuller view of the structure universe. I think these are pretty important applications, at least in my view. As to whether this will help drug discovery immediately, I think it's perhaps less clear-cut. I think there will definitely be future applications that are very impactful in drug discovery, but so far it seems like people haven't— it's only been a year, so it's a really short time, but so far it's not as clear how it would help drug discovery in the traditional sense yet. Because for a lot of the important drug targets, people often already have solved crystal structures. But I'm sure with AlphaFold and with other newer advances in machine learning, we can perhaps discover new targets and expand the conception of what drug discovery should look like. So here is a video for the nuclear pore. This is imaged under cryo-EM and solved with the help of AlphaFold. And you can see, I'll play this again, the pore can dynamically expand and contract. Each different color in this video represents a different protein chain, so it's a pretty big thing with a lot of different protein chains. And to give a more visual sense of the data scale for protein sequences, so sequencing in biology, most of data comes from sequencing because DNA sequencing is one thing that we know how to do very reliably and now cheaply. So for all kinds of biological experiments or measurements, usually people try to tie it in one way or another back to sequencing because that's one of the few ways we know how to generate large quantities of data. Maybe imaging, microscopic imaging, could be another one, but Really, for all kinds of biological measurements, usually it's back to sequencing. So sequencing is pretty easy to do, and there are billions of known protein sequences from all kinds of organisms or from all kinds of places. And compared to these billions of known protein sequences, there are actually very few experimentally determined structures. It covers I would estimate around 0.1% of the entire sequence space, and it has a lot of biases, so it's not an even sampling of the sequence space, but rather some particular corners of the sequence space. Now with the new data generated from predicted structures, we're covering around 200 million And well, maybe this is not the best picture for it, and these 200 million are chosen more uniformly across the sequence space. So it's not just the absolute number that matters, but also that with new predictive structures, we're able to get a more uniform or broader picture of the sequence space. Okay, maybe I'll pause again here to see if there are any questions. And since we're switching topic to protein sequence models, Any questions?

So this is, I'm sure, a super naive question, but thinking about like trying to synthesize proteins to do different things, it feels like a super hard task to like try to design structures by figuring out locations of all the like, you know, microscopic elements. So I'm curious, you know, like, like, you know, if I'm building like something I like, I don't have to solve for like the molecules in wood or something like that. Like there's sort of higher level abstractions that I know are like good substrates to use and tweak and stuff like that. So I guess I'm curious like to what degree you're learning like sort of reusable motifs versus like every new design is like a new problem where we have to solve for all the microscopics, if that makes sense.

Yeah, that's a great question too. I have a slide here that might be— yeah, maybe I'll skip ahead a little bit to generative models. So here is a video of a machine learning diffusion model generating a protein structure. So it, I guess, it already starts with some parts of the structure, but it's able to actually predict all the remaining— the positions of all the remaining amino acids to somewhat reasonable accuracy. So I guess to the first question, it is possible to do it directly without the abstraction today with diffusion models and also with some other hallucination models. So there are ways. I think we are at the scale of protein structure data that we can learn it amino acid by amino acid with all of the abstraction. So this is on the machine learning side, but on the protein design side, when we actually make the proteins in lab, I would say today, or yeah, I would say today, or at least a couple years ago, when people make synthesized experimental proteins in lab, it's usually still more basic building blocks. As you can see here, there are some common motifs like the alpha helices, and in particular alpha helices are the helical structures. Remember we earlier talked about there are these helical structures or the beta sheet structures. The helical structures are usually more stable and easier to design de novo than the sheet structures. So if you look at the design proteins, often they have a lot of these helices. So I would say the machine learning predictive power is getting there to do the full amino acid by amino acid modeling, but based on currently what people have synthesized so far, it's still smaller proteins and with a lot of these motifs.

I think there was another question.

So I was wondering, for new protein structure prediction, how confident can you be in the model predictions? And yeah, how can you say the model is right? Yeah, that's a great question. I would say one of the reasons why people use AlphaFold 2 a lot is that it comes with a confidence score, and the confidence score is pretty well calibrated. To me, that was a pretty surprising thing because I used to not really trust machine learning model confidence scores or these, I guess, uncertainty measurements. But based on the evidence so far, it seems like the confidence score that AlphaFold produces is pretty well calibrated, and it seems to know where it's confident and where it's not.

Any other questions?

Well, thank you. Okay, so winding back, so we talked about protein structure prediction. Beyond protein structure prediction, a lot of, a lot of the recent research activities are also happening in protein sequence modeling. So the goal of protein sequence modeling is to directly model the distribution of protein sequences without reasoning about 3D structures. And part of why this might be useful in cases is that not all proteins have a well-defined 3D structure. There are also intrinsically disordered proteins that might be up to, let's say, 30 to 50% of the human proteome that don't necessarily come with any protein structure. So that's why when we think about proteins, it's not just the structure that matters. The protein sequence modeling is also an equally important part in this.. And there has been over the past 2-3 years a lot of work in scaling up deep learning language models for modeling protein sequences. So this is the plot here, the scaling plot here is from a recent paper last month. I think this might be the biggest protein sequence language model that people trained so far, although it's all happening so fast and there are a lot of industry players in this, so it changes probably every month. So we can see that in this example, and also in general, the consensus so far seems to be that larger protein sequence language models do seem to do better. We haven't really hit a scale ceiling yet. So there are a number of industry groups working on scaling these models up more and more. So, I guess when we talk about scaling and the notion that scaling works here, it's in this direct measurement of perplexity. Like, if we scale up the model more, then that achieves a lower perplexity, and that means the model seems to model the distribution better. But it hasn't quite translated into direct applications. I would say that usually the way people test protein language models is that there's the raw perplexity measurement that's on a held-out set. That's pretty straightforward. And then there are these downstream tasks or zero-shot prediction tasks where people take the trained large language model and then apply them for a bunch of downstream zero-shot prediction task. And it seems like even though scaling does provide a lot of benefit for certain things, overall for some of the, or maybe a lot of the downstream tasks, we see that there's maybe some kind of inherent ceiling for the performance on the downstream tasks, and we don't really see that much better and much improved performance through scaling. Part of that might be because there is some inherent noise in the data measurement, and if there is noise in the data measurement and the datasets are not perfect to begin with, then maybe it's harder to see the model improvements in those downstream tasks. Or it could be that there is some kind of, not necessarily distribution shift, but, or maybe it's also a form of distribution shift, like the things that the large language models are trying to capture might not exactly be aligned with what those datasets are measuring exactly. So because of these, I would say that scaling on protein language models seems promising so far by certain measurements, but for some of the downstream tasks such as fitness prediction that people care about, there has been limited, pretty limited improvements. The main use cases for protein sequence models is one is to predict effects of genetic mutations. I guess even though genetic mutations happen at the nucleotide level, but because we can translate the nucleotides into protein sequences, we can then interpret the mutational effects in terms of how it affects the protein function. That's one area in human health where these models could be applied to identify and interpret effects of genetic mutations. Another way these protein sequence models could be used is to guide sequence design choices in protein engineering. In protein engineering, there are multiple ways to design new proteins. We discussed earlier briefly about designing entirely new proteins, but I would say the more common practice today is to start with a protein that already almost does what we want to do, and then start with that to modify parts of it, to do directed evolution on it, or to mutate certain parts to make it better. And with these sequence models, we can then perhaps start with a sequence that already does something close to what we wanted to do and then use sequence models to generate nearby sequences that might be better or functional in different ways.

We got a quick question.

I just wanted to make sure, and sorry if I missed it, but the modeling there is just next amino acid prediction?

Yeah, that's right. So it depends, I guess it's pretty similar to NLP when people could do next amino acid prediction in an autoregressive way, or it could be masked token prediction in a BERT style, or it could be— I think people have also applied other losses or training objectives. So I would say anything that could be done on natural language could also be used for training here.

What's the size of the dataset? In this plot, do you know?

Yeah, it's billions of sequences.

Billions of sequences. Okay, cool.

Thanks. Right, okay. So yeah, as an example of how it could be used for scoring genetic mutations, we could start with a sequence and then score the sequence likelihood and then and also score the sequence likelihood with mutations included to compare the sequence likelihood. And usually on average, the assumption is that higher likelihood sequences are supposedly better. And in the protein engineering application, there is usually a feedback loop or cycle. Where we start from something that kind of already is close and then generate a diverse library of sequences that are related but different. And these are candidate sequences that we believe could be good or could work. And then there would be a screening round to gather the experimental data on these set of library sequences. And then as a feedback loop, we can further design new sequences to go through this. So it's kind of similar to the active learning framework, but one key difference here is that usually biologists won't test one sequence at a time, but rather it's a lot more efficient to test a large batch of sequences at a time. These could be millions or tens of millions of sequences in one round of testing. So there's also an interesting, perhaps batched active learning problem here of how do you actually do a good library design to hedge the bets and then most efficiently gather information about the sequences. So there has been over the recent years some work on formulating this protein engineering problem as an active learning problem and proposing algorithms accordingly. Okay. Yeah. And then one interesting question I always think about for protein language modeling is, so in natural language processing, when we think about models like GPT, it's interactive in the sense that we can give prompts to the model that we understand as humans and interact with the models this way. And that has enabled a lot of the applications for the language models. But for the protein sequence language models, it seems that we even— researchers don't really understand the protein language necessarily. So the applications are perhaps— there could be perhaps even more applications if we find a way to somehow more cleverly interact with the models or prompt the models. This would maybe require different ways to train the model or represent the data. I haven't quite figured out what make sense there, but that's one thing I often think about. How do we make these models more interactive, or ways to expand the models into different tasks other than just making mutations? All right, so we've seen this. Yeah, and then in parallel to the protein sequence language models that can generate sequences, there are also these structural generative models that can generate new structures for protein design. This one that we just looked at is a denoising diffusion model, and our own work from my internship last year at Meta AI is also looking at generative models for protein design where we start with a target protein 3D structure and then design sequences for the structure. So for example, this could be combined with the diffusion model that maybe generate the structure or parts of the structure first, and then we can generate a sequence to satisfy the structure. There are also many other use cases for this class of inverse protein models that we worked on. We covered protein design already. Then I want to talk a bit more about how these machine learning algorithms work. We've talked about the Protein Data Bank data that's very important for enabling machine learning applications. And another, in addition to Protein Data Bank, another very important source of data for biological applications is evolutionary data. So when we think about evolution, this is represented as an evolution tree where there are common ancestors for different branches of life, and then there's a whole tree of evolution. And what evolutionary data means here is that we measure sequences, the genetic sequences observed for different organisms in this whole tree. Well, except that one problem is that we can't actually observe the whole tree because the common ancestors might already be extinct, so we can't measure them. We got a snapshot of the evolution by observing what's still alive today. This is actually very rich information because evolution is this whole billions of years of process of exploring a lot of things that might work or might not work, and then the good things that worked survived. So it's pretty rich data even though it's not labeled necessarily, but from observing this whole distribution of what sequences survive through evolution, we can learn a lot about protein and also beyond proteins, other areas of biology. So as an example, there is this helicase protein that's very important in basic biological function. And on the left, it's a copy of the helicase protein in humans. So everyone here, we all have this protein. On the right-hand side, it's the helicase protein in thermophilic bacteria. So if you go to Yellowstone National Park and see the hot springs, you can find this protein there. And if you look at them, they're actually very similar. And this is through millions or maybe billions of years of evolution from the bacteria to human, but actually the protein structure is very much still roughly the same. However, even though the structure looks pretty similar, the protein sequence has evolved to be very different. There are only about 30% of amino acids that are shared between the bacteria and human. So here, this is real, the sequences here are real. There's the human version of the helicase protein and the bacterial version of it. And you can see that most of the positions have mutated away, but there are 30% of the positions that stay the same. So from a machine learning perspective, what could we learn from these mutations? Well, one easy thing to conclude from looking at this is that the 30% positions that did not mutate are probably very important to be conserved for the protein structure and the protein function. The, the reason that they stay the same is probably because if they had changed, then the bacteria would have died, or we would have died. So by looking at what's conserved and what's not conserved, we can deduce a lot of information about what's critical for the protein function and so on.. And this alignment is usually called a multiple sequence alignment. Here I'm only showing two sequences, but in a lot of other use cases, there could be thousands or hundreds of thousands of sequences in this multiple sequence alignment matrix. And if you think about what we could do on the data, one thing is that we could look at the learned sequence density on these sequences. For example, here we see some clusters of proteins may be closer together in clusters, but there are some further sparse areas. And the higher density areas where we're more likely to observe sequences means that they are more likely to contain functional proteins. Even before deep learning came about, there were a lot of simple models on evolution there that also work pretty well. For example, HMMs or these Markov Random Fields pairwise linear models. They all function, I would say, in the same principle as the deep learning methods today, which is to model the distribution of sequences and have the assumption that higher sequence likelihood through evolution means means more likely functional proteins. Today, instead of the HMMs or Markov random fields, we can now also replace them with deep learning models. And the main advantage of doing so, as opposed to using the HMMs, is that there is less restriction on the protein length, or there's no need to enforce this multiple sequence alignment We can have the same model learn the whole sequence distribution regardless of the sequence length or the alignment, and this allows us to do the same model on a lot of different families of proteins together. It may be in reality a mixture model with— I guess one way to think about it is that it could be a mixture model of a lot of HMMs, but, or maybe it could learn some generalized rules from all these families together at the same time. So that was for protein sequence modeling, but even in structure prediction in AlphaFold 2, it relies pretty heavily on the evolutionary data. So as the input of AlphaFold 2, it looks at this multiple sequence alignment alignment, for example, the same one shown here. So this would be the input of AlphaFold 2 if we want to predict the structure of this helicase. And by looking at the multiple sequence alignment as input, the model can then extract structural information from the multiple sequence alignment. And this evolutionary information is very critical for AlphaFold, also based on studies Essentially, if we don't feed in this evolutionary information to AlphaFold and only give a single sequence, then it works much, much worse. So the— yeah, I would say in a lot of different protein machine learning applications, evolutionary information is really a foundation for a lot of the models. But it's for AlphaFold 2, it's not only the evolutionary information, it also uses structure modules to iteratively refine the structure even after extracting the evolutionary information. So the evolutionary information is not perfect, and then the model also tries to learn some physics principles to refine the structure. Here's an example of how it iterated. So I'll start at the beginning. So the model has two stages. First, it already does this Evoformer to look at the evolutionary information, and afterwards it has an iterative refinement on the structure. So this is the iterative refinement part. You can see that initially the predictions are not quite right, but through iterative refinement it's able to adjust it better and better. Initially there are some more global adjustments, but then towards the end it's more local adjustments. I think that this was a main innovation from AlphaFold for doing the recycling iterative refinement. And it turns out that perhaps it's much easier for the model to learn how to iteratively refine a structure rather than directly predict the right one right away. And often it takes dozens of iterative refinements for the model to get to a good structure. Sometimes in more complex proteins, it might be even more cycles of iterative refinement. This is the evaluation study on AlphaFold. You can see that a lot of the other things maybe don't matter so much, but the recycling, which is iterative refinement, is pretty important. And there are also some other innovations that are pretty important in in terms of how they do the attention and the structure module, but I won't be able to get to the details of those today.

Can I ask? I assume the multiple species alignment is between different species. Do people do alignments, you know, like if I have like Hox family or some repeated gene in a single species, I could imagine doing alignments of the common proteins.

Yeah, I think people also do. So yeah, there's multiple sequence alignment across species, but also let's say if I want to study finer-grained polymorphism within the human population, then it's also the same idea. Also makes sense to do the alignment within human sequences.

That makes sense. But I guess you could also do it within it, like even within a single human, I guess.

Right, where like— yeah, yeah. So for a single human, usually we— for example, for the helicase protein, there's only one copy. So for a single human, the protein would all be the same. But for some other things, like people study cancer mutations, because in cancer a lot of the genetic mutations are happening, or in immune systems where there is a lot of mutation, then that could could make sense within a single human.

Sorry, I can let you go and keep talking for a second, but even when we say like, oh, there's only a single protein, I guess I'm curious, you know, like, to what degree is like the history of evolution within like a human, like, you know, if I have two proteins which maybe we call different now, but they came from like duplicating and then modifying some common ancestor or something like that, is there like like, yeah, can I use this like sort of common evolutionary history to like extract more conservation information, or is that just like not that much of those examples?

I haven't seen that being used, but maybe in some cases it might make sense.

Yeah.

Any other questions?

Yeah.

Oh, what do you mean by recycling in AlphaFold 2's architecture?

Oh, so, uh, okay, it doesn't show here within the structure module. So here, um, do you see recycling at the bottom? So it— after making one prediction of the structure, the structure then is also fed as an input to the model model to get a second iteration of the structure, and then again. So there's the MSA input that's always the same, and then through the model there is one candidate structure, and then this candidate structure and also the confidence scores get fed back into the model as input.

Okay, got it. Thank you. Also have another question: how can we generate an evolutionary set of mutated sequences? Would that involve using, say, like multiple sequence alignment, like the MSC Transformer, or maybe inverse folding, could they be used for this?

Uh, could you clarify a little bit?

Would, um, maybe like, for example, like generating like an evolutionary set of mutated sequences. For example, if I've got like the older sequences and then like what could be the next variant if it's—

Yeah, um, so they're— yeah, they're actually— yeah, these models could be used for doing that based on taking gradient enhancing the likelihood. And there was a pretty interesting paper last year called Evolutionary Velocity that uses language models to extract or map out the evolutionary history of viruses.

Okay, got it. Thank you. Sorry, can I ask another question? Is there any way to detect mutation hotspots given if you've got, say, like an evolutionary set of sequences?

Yeah, from multiple sequence alignment, for example, we can see which positions are mutated most frequently. And then for new proteins that we don't have multiple sequence alignments for, these language models or structure-based language models, such as the inverse body models, can also be used to score where the mutational hotspots are. Okay, got it.

And we could just mutate the spots further, say, using like masked language modeling or span masking. Yeah. One last question, I'm so sorry, but how can we evaluate the mutations for their fitness prediction that you mentioned?

So for the— if we're using a model to evaluate the mutations, then it would be to look at the likelihood of the sequence under the model with or without mutation. And if we're doing that in lab, then people synthesize protein sequences with those mutations and then measure that together with a control in lab.

Got it.

Thank you. Well, thank you. Okay, so in the interest of time, I think we're already over time.

We have about 10 minutes left in the slot. Maybe I'll also let you pause for future questions whenever you want to. To use the 10 minutes as you like.

Okay, thank you. No, that's perfect. I have the last section. I think 10 minutes is just about the right time. So we talked about how some of those models work and the applications, and I wanted to end with two maybe more open-ended questions for all of us to think about. One is this question of search and learn versus learning a universal model. So we've seen that in the case of using a multiple sequence alignment, we are looking at the whole database of evolutionary sequences but then only grabbing the ones that are relevant for the target protein that we want to predict structures for or want to model. The visuals here are showing, let's say we have this gray circle is the billions of sequences that we have in the whole database. Now in the case of AlphaFold or other MSA-based method, we do the search first and grab a small set of hundreds or thousands of related sequences, and then we learn a local model in some sense based on these hundreds or thousands of sequences. And in contrast to this search and then learn approach, the large language model approach tries to learn a universal protein language model on this whole dataset. And then on top of that learned representation, it could also be used for doing structure prediction. There is some recent work this year after AlphaFold 2 was published on using these language models without any multiple sequence alignments for doing structure prediction. It's still hard to say how the results compare exactly, but there has been a lot of recent progress on that as well. So there is an interesting question of which one is the more efficient way to go about this, is it better to search and then learn a local model, or is it better to just have the model learn and reason about the whole thing? It's still not very clear to me. In terms of the computational time, the search actually is often a computational bottleneck. For example, in AlphaFold, or usually any MSA, multiple sequence alignment-based method, the search to grab these related multiple sequences is actually the main computational bottleneck and takes most of the time. In the protein language modeling approach, since it gets to skip this search part, the inference time is much faster because it doesn't need to do the search. But maybe in compensation for that, the training time might take longer, but it largely reduces the inference time and makes it faster to use. So I think that's an interesting comparison and contrast between the two. So far, I guess the best performing models are still MSA-based, but it does have limitations and the search is slow. The other comparison or question that people often think about is whether these language models are effectively also doing search. I guess in a way, these language models have— let's say the model already has billions of parameters and then they're trained on these databases of billions of sequences, then maybe the model can somehow actually memorize a lot of these sequences and then implicitly do a search internally in the model. There's, I guess, within the biology, the machine learning for biology community, people haven't really crystallized this or tested this in a way. But maybe for people who have thought about this more in other contexts, I would be curious to know if you have any idea of how to test this this hypothesis of is this model implicitly doing the search, and then maybe it also has some kind of implicit MSA while doing the reasoning for doing the structure prediction or other tasks, or is it actually learning something completely different and we can't quite think about it in this memorization and search paradigm. So, I would say to me these are both most open questions that a lot of people think about or have hypotheses or guesses for, but they seem, at least for now, hard to test in a rigorous way and examine. Okay, yeah, that's about it. So I'll end here and see if there are any more questions.

Thank you. Thanks. We have some time for questions.

Jamie?

Is there any similarity between the iterative refinement that you showed that AlphaFold takes in approaching the final structure and the actual folding process that takes place physically in time with the protein? No, I don't think so. Yeah, I don't think a protein could start with this aggregated part. Yeah, but maybe towards the end it might model some of the physical forces on the protein. So yeah, there— at least there hasn't been any evidence that it follows the folding process. Should we understand the physical process as a sort of like energy minimization? So yeah, maybe, yeah, a good— that's a good question. I think I'm not super familiar with how protein folding actually follows an energy minimization or not. So maybe I'm not in the best position to answer this. I think there's some aspect of energy minimization in it, but in protein folding in cells, there— it comes out of a ribosome, which is another protein, so it might also be affected by other things that's not just purely the energy of the protein itself.

Any other questions? I had a sort of basic question. Do you know if we— when we train these sequence models just on the protein sequences, do we know if the representations that they learn are meaningful in, uh, for any sort of higher structure? Like, do we know if the sequence models sort of cluster proteins by structure or function or something like that?

Yeah, that's a good question. For the protein sequence models, they actually— yeah, it's pretty interesting. They are— they can also be used to do protein structure prediction even without further training, but that the performance is not super good. And in the existing papers, it's people look at the attention maps for the protein sequence models, and then the attention map attention maps seem to correspond to protein structure. So I'm not sure as sure about the representation for the sequences themselves, but it seems like at least in the attention map that shows the model does do some level of structural reasoning.

One more question.

Yeah. Curious about whether people augment the next amino acid prediction with like just throw in as much extra info as you can, like conservation and structure. You can imagine just like doing that all as the next token prediction.

I haven't seen any work doing that, but that could be interesting. Yeah, right now it's usually, at least like what people have done so far, is usually just on protein sequences themselves. Although I guess maybe one contrast is the inverse folding model here starts with the protein structure and then models the sequences through next amino acid prediction. So when we compare this with just doing the next amino acid prediction without the structure as input to the encoder, The performance is much, much better with the structure. The perplexity here is around 4, and in contrast, if we only look at the sequence without the structure, the perplexity is usually around 11 or 10. So from 10 to 4, it's a pretty big reduction in perplexity. The structure does give a lot of information.

One more question.

Hi, yeah, great talk, thank you. So when the polypeptide chain comes out of the ribosome, in many cases there are other proteins, chaperones, that help the protein folding. So I was wondering if incorporating also the chaperones structure in the model could help. I don't know, is this related with the previous question, with one of the previous questions?

Yeah, I think that's a really great idea. I haven't thought about that, but I like this idea a lot. And maybe during the recycling, perhaps rather than just doing the recycling on its own, we could include the chaperones during this recycling somehow and jointly model them. I think that's a great idea.

Thanks a lot for the questions. We're out of time now, so let's thank our speaker again.

Thank you. Yeah, it's a lot more fun with all the questions, so thank you all for all the questions, and thank you for your time.

Thanks, and thanks everyone for showing up to this Simons Week in summer school. Invite Peter up here to wrap us up.

Yeah, let me add my thanks. Thanks everyone for being here. It's been a lot of really interesting discussions throughout the week. And a big thank you to all the, all the speakers we've had. Amazing lineup, and they did a fantastic job. And especially thank you to the organisers. So Nati Sribro, Surya Gonasika, and especially Pritam, Youngtak, and Spencer. Hugely energetic, hugely thoughtful in putting together the program for this week, the tutorial talks, the research talks. It's all been really fantastic, and I've been sort of in the background seeing the discussions have been going on. And, you know, these folks really were the driving force behind this week. I think it was really excellent. So big thank you to all of those.