Founder Stories: Alexander Rives, Evolutionary Scale Language Models

Today, I'm going to talk about our work dating back a number of years now, I guess, looking at training large-scale language models for biology and specifically for proteins. So, okay, there we go. And so I guess this is a room of biologists and computational biologists. So these are things that I don't necessarily need to say, but I think it's really worth pointing out the importance of proteins and really the potential for being able to understand them, design them, and really make them programmable in the way that other systems that we create are programmable. So the panel on the left shows an example of a checkpoint inhibitor. So this has been really a revolutionary treatment for cancer. It's a protein which blocks the ability of the immune system of tumors to evade the immune system. The center panel depicts a protein which degrades plastic. And so, you know, nature has invented all kinds of really interesting and amazing and powerful and beautiful functions. And one of them actually is a protein that has evolved to break down the bonds in plastic. And so being able to really search evolutionary space and find interesting proteins has implications for a variety of green technologies. And then the panel on the right-hand side depicts RuBisCO, which is one of the most abundant enzymes on Earth. And this enzyme is critical, of course, in carbon fixation, and engineering it is an important target for climate applications. So this just shows some of the possibilities if we could design proteins. So I think we're all really aware of this kind of transformation that's happening in generative AI right now. And there's a really, I think, exciting opportunity in biology. And of course, biology in some ways is much harder because you know, we have to collect data around biology, but we've had this enormous decrease in the cost of gene sequencing. And this has led to an exponential growth in the amount of biological sequence data. And of course, we're having these incredibly rapid improvements in compute through parallel architectures. And so, you know, every 2 years and now every year really, we're getting GPUs that have, 2x the throughput or more. And so this is really enabling, you know, this line of research, which we mostly seen in generative AI for language models, for kind of classical NLP and vision where it's possible to scale these models up to hundreds of billions or trillions of parameters. And then finally in biology, and I think a really important enabler for kind of the future of generative AI in biology is that we have an increasing ability to synthesize genes and to actually create the proteins that we design in silico. So this is the perspective we take. It's a very basic perspective, of course, which is the idea that the sequences are codes for proteins through the physics of folding. Sequences specify biological structures, and those structures encode the function of proteins. And the idea here really is, and, you know, there's kind of a very beautiful picture here, which is, you know, nature has created this machine, the ribosome, that's kind of this molecular factory, and you can put in you know, the code for any molecular machine into this factory and it will produce it for you. And this is actually extremely scalable. And so, you know, we can make biological molecules at scale through this. But, you know, what we're really missing is an understanding of how to write those codes. And so we have, you know, as I talked in the previous slide, we have this technology that allows us to read gene sequences and we have a gene sequence printer and we can put that, we can put those printed sequences into the cell and give them to this machine as instructions to manufacture proteins. But what we really don't have a deep understanding of is kind of the fundamental kind of coding language that these molecules are written in. So the question is, could artificial intelligence be a way to really develop an understanding of that. And so the approach that we're going to take here is really this very simple approach. It's the same approach that's being used to train language models for text and video prediction models and a number of different types of generative AI applications. And it's an idea that's called self-supervision. And basically, the idea is really learning learning from the patterns in the data, learning to reconstruct the data. And so the approach that's mainly used in language modeling, as we all know, is this autoregressive approach. So it's to be or not to be, that is the blank. And so the model kind of fills in the blank based on its knowledge, based on the information in the previous words. And so you predict each token one after the other. The models that we are training use masked language modeling. And so the idea here basically is that you're going to blank out a number of positions in the sequence and you're going to ask the model to predict those positions given the surrounding context. So to blank or not to blank, that is the blank. It's very easy for us to fill in the words in that soliloquy. And I think if you, So this is a very simple thing, incredibly simple thing, but maybe to make a connection to literature, Borges had the idea of an infinite library that was full of every book that had been written and every book that could ever be be written. And sort of the question I think you need to ask, and I'm saying this because I think it's useful for really thinking about what we could learn from biological sequences, is if you had a model and it had to compress every sentence that had ever been written and that could ever be written, the idea is that within that model there would have to really materialize in an understanding of the underlying reality that's giving rise to the patterns in those words. And so we can think about that parallel for biology. And this concept is really known as the distributional hypothesis. And it dates back to the '50s to a paper by Harrison. Basically, what it says is that the meaning of of words, of language can really be inferred from patterns across a large corpus of texts because the meaning constrains how we use words. And so it's imprinted into the text, imprinted in pattern into the text. And so there is this very beautiful parallel with biology. And I think the way to think about that parallel is sequences are generated by random mutation, and they have to pass through the sieve of evolution. And only the sequences that are consistent with biological structure and function are able to pass through that sieve. And so when we conduct large surveys of genetic diversity, of protein diversity across all of nature, you know, we're seeing this vast text in which are patterned those deep constraints that are acting through evolution. And this is an idea that goes back, you know, really almost to the beginning of gene sequencing and molecular biology. And, you know, one of the things that we know is that as you, when you generate a multiple sequence alignment, you find a set of evolutionarily related proteins, you know, you can really look at the patterns within that family. And so important functional sites are reflected through conservation, a single amino acid, which frequently or always occurs at a given position in that protein family. And then the other thing is really coevolution. And so if you have, you know, the three-dimensional structure of a protein, you have, you know, two positions, I and J, which are are far separated in sequence space, and those positions come back together in the 3D structure of the protein, then evolution isn't free to choose the identity of the amino acid at the first position, at position i, without changing the identity at j. And so this really creates these dependencies that can be observed in, sequence patterns in a protein family. And so this has really been the basis of protein structure prediction, starting with kind of the, you know, simple pairwise models. Here are the POTS models that are shown in this slide and really extending to AlphaFold, which uses the same principle. And so the input, I think as we probably all know here, to AlphaFold is in a multiple sequence alignment. And so AlphaFold is really inferring the structure from the patterns within that multiple sequence alignment. So this is a really different approach to extracting that evolutionary information and those evolutionary patterns. And, you know, so the basic idea is, as I said before, is we're going to train a transformer model on protein sequences across, you know, all of the tree of life across vast evolutionary diversity. And those models are just going to be trained to predict amino acids that have been masked from sequences across all of nature. And then what we're really interested in is in learning to perform this very simple task, this pattern completion task. Is there kind of deeper biological knowledge or information that emerges within the parameters of the model? And how can we use that information? And so one of the first things that we saw is that there is this interpretable picture of protein structure that actually emerges within the representations of these transformer models. So the transformer is basically a neural network, which takes a set of discrete tokens, it transforms those, into high-dimensional vectors, one vector for each position in the protein sequence. And the key kind of mechanism within the transformer is something that's called self-attention, which is really an operation which basically, I think really the best way to have intuition about it is it's an operation that learns to synthesize a graph. And so what it does is each position in the sequence outputs a key and a query. And those keys and queries interact and positions through a dot product. And so positions which interact strongly, then a value is basically summed according to the weight of that dot product. And so you can really think about this as those attention weights forming a graph. And so this is depicted on the slide on the left, basically an adjacency matrix, a weighted adjacency matrix. And then information from each node will be summed across those edges. And so you can just look at that graph basically. And so that's what's shown on the right-hand side here. And so the bottom depicts, so I guess maybe from bottom to top is basically the input to the transformer all the way to the output layers. And then each layer has 20 attention heads within each layer. And so each of those blocks is basically showing the activation of one attention head. And so you can just correlate those attention heads with context in the 3D structure of the protein. And so that's what's shown here. If you just use the activation of that attention head to rank all the potential pairs, and then you look at the ability for that ranking to predict whether there's a long-range contact within the protein structure, then you see that there are these heads which really directly correspond to contacts in the 3D structure. And what's kind of interesting about them too is that they're kind of deep in the network. So something that emerges, you know, deeper through the processing of the model. And you can look at this visually and, you know, even, visualizing one single attention head, you can kind of see what the structure looks like. This is now showing if you fit a regression, it's a really simple regression. There's 660 attention heads in this model. So we fit 660 plus 1 bias parameter. And so this simple logistic regression is basically going to take the activation of each attention head for a given pair of positions in the protein. And it's going to output the probability that there's a contact. And so you can actually get a state-of-the-art unsupervised contact predictor from this. That's just, you know, you basically project it out of the model in this way. And this can be fit with just a few training structures. So this is showing that, Basically, as you add a few more training structures, that gives more signal to pull out which attention heads are relevant to predicting the structure. And basically, at the point that you use 20 structures, you've saturated what's possible here. And the way to think about this, of course, is that you're really just using those structures basically to identify which of those attention heads correspond. To contacts. So the other thing that we observed, and probably unsurprisingly because this model's kind of learning probabilities of amino acids across proteins across nature, is that the model learns about function and the probabilities that it predicts at each position in the sequence correspond to function. So this is an example. Of what this looks like. So this is a DNA methylase. And what we've done is just ranked all the positions by the model's entropy at a given position. And so, you know, if the model's very confident in the amino acid and what it thinks the amino acid assignment should be at a given position, then it would be low entropy. And so, you can see here the top 10 positions for the bottom 10 positions for entropy. And you can see that they really cluster within the active site of this protein for the most part. And the lowest entropy positions are the ones that actually interact with the substrate. And you can also look at this more systematically. And so this slide is showing a set of deep mutational scanning experiments. So each experiment is down on the x-axis basically. And for those who aren't familiar with deep mutational scanning, you know, it basically gives a measurement of the effect of a variant or mutation on the function of the protein. And you can do this at scale. So there are experiments that have tens of thousands or more mutations that are all simultaneously measured. And so you can ask the question then systematically, does the model's kind of understanding of sequence correspond to the mutational landscape for these proteins? And so, you know, very simple scoring function, basically taking the log odds between the mutated, the probability assigned to the mutated position by the model versus the wild type amino acid assignment. And so if you just rank everything by that probability, you can calculate a Spearman best guess row against the actual mutational scanning data. And we were able to find that, you know, these language models basically produce state-of-the-art mutational effect predictions. And what's really interesting about this is it's really fully unsupervised and it's also zero-shot. So after you've kind of performed training at scale, you can input a new protein sequence and you don't need to perform more optimization or learning or gradient steps. You know, the model's just inferring the landscape from what it has learned. And I want to highlight some work. This is not from our team. This is actually totally independent work by Brandis et al. in Vassilis Netronis's group. And they were actually able then to employ this approach to score the clinical effect of mutations in protein sequence. And they found that just directly using ESM1B here produces state-of-the-art predictions of the clinical effects of mutations. So it's a really cool paper if you want to check it out. So turning back to structure, I guess it's now the summer before last, we performed a scaling study on these protein language models. And we created a family of models, ESM2, from 8 million to 15 billion parameters. And so this plot is really showing what happens as you scale these models up. And the value on the y-axis here is the long-range precision at L, and it's calculated using that same, you know, linear projection that I showed before. So we're just really kind of looking at how much, you know, kind of low-resolution structural information materializes within the model as we scale it up. And I think there's two things to call attention to here. So the first, both of which are quite obvious. The first is that there's this frontier. And so each line is a model of a different scale. And so as you scale the models up, you see, you know, an improvement that continues with scale across all depths.. And the x-axis basically showing how much evolutionary information is available in training around each of the proteins that we're evaluating on. So what's the number of MMseqs2 hits for that protein in the training set? And this really kind of corresponds to the depth of the multiple sequence alignment that you could generate for each of those proteins. And so the other thing that we see is that there is this dependence on depth. And so the model, where there's more evolutionary information, the model is capturing a lot more information about the structure. But we see that even for the proteins where there's very little information, this improvement with scale still applies. And so for ESM2, we really wanted to take a step further and look at atomic level structure in these models. And so what we found is that you can actually project out the full coordinates of every single atom in the protein from these representations. And so here, basically what we're doing is we're taking a frozen model, we are fitting an equivariant transformer to that. So we're using the same architecture that AlphaFold uses with invariant point attention. And then we're going to supervise that projection with experimentally determined structures. And so what we can find, and there's no folding blocks or anything else here. So it's really just pulling out the information in that last, layer of representations in the model. And so what you can see here is that this is an example from a temporally held out test set of chemioproteins that as we increase the scale of the model, you know, more and more kind of atomic level information emerges to the point that you can get a very good, low RMSD structure prediction just through this projection. And so we were able to use this then to create a method for structure prediction that really makes its inferences from an individual sequence. And so this is the key distinction from AlphaFold, of course, is AlphaFold has to, the first step is kind of performing a search constructing a multiple sequence alignment. And it's that multiple sequence alignment, which is the input into the model. And, you know, this allows for very accurate predictions. It comes at the cost of, yeah, of speed basically. And so if you're, the state-of-the-art MSA search pipelines can take, you know, minutes or, or more than 10 minutes. There are faster, less sensitive methods that have been developed that can greatly reduce that time. But it's still a very significant step and really critical for AlphaFold to work. And then second, the network itself has layers which have to process the multiple sequence alignments. So that adds computational cost. And so really what we're able to do is replace that multiple sequence alignment search step with a forward pass of the language model and just directly compute these representations. And then, so using this frozen language model, essentially we can feed those in to a folding truck and then a structure module and generate really high-resolution 3D structure predictions. And this is what it looks like. So the accuracy approaches that or really matches that of AlphaFold for many proteins on the Cameo test set. We're a little bit better than RosettaFold, a little bit worse than AlphaFold on CASP14. There's more of a, gap. And then the right-hand side of those plots, and maybe one other thing to point out here is that these test sets are temporally held out. So the models have been trained on data before a temporal cutoff. And then the predictions are on held-out CAEMEO and CASP proteins that were, where those structures were submitted after the temporal holdouts. And so if you perform the ablation where RosettaFold or AlphaFold, you remove the MSA and you use them in a single sequence mode, you see that there's a really big degradation in performance. And obviously AlphaFold and RosettaFold haven't been trained in this way, so that's out of distribution for them. So you wouldn't expect them to perform well in that setting. But this setting is relevant for protein design where there's interest in really putting a model basically within the inner loop of a design algorithm. And there, you really need to design the individual single sequence. So having a fast and highly accurate structure prediction method is really critical there. And similarly to AlphaFold, the model has this in-context learning of protein-protein interactions. So even though we supervise on single chains, We can make predictions for complexes. And another key thing for kind of using a method at scale is that the predicted error correlates with the actual error. So that's really just showing that, you know, if you use this in practice, the model's going to output an estimate of its error, and that estimate correlates with the actual error. And so you can use the model's estimate of its error to judge how good the structure prediction is likely to be. So in, I guess about a year ago now, we released the ESM Metagenomic Atlas, and that's shown here. And ESMFold is 60x faster in inference than AlphaFold, up to 60x faster. And of course, it removes that MSA generation step. Generate structure predictions at scale with this approach. And so we predicted structures for over 617 million proteins. We worked actually with the Magnify team to expand this. So there's now over 700 million proteins in this atlas. And this is the largest database of predicted protein structures to date. It was 3x larger than AlphaFold DB at the time. And it covers essentially all of Magnify, which is a metagenomic resource created by EBI. And these are, of course, some of the least understood proteins in the world, and we know very little about their structure and function. And so this is really the first survey at scale of these structures. All of this is available on a website, ESM Atlas. .com, you can go to and you can search by sequence and by structure. You can also use ESMFold to fold proteins. Okay, I'm going to switch topics now to talk about generalization and using these models to design proteins. So really using these models generatively. So far I've really been talking about representation learning and kind of understanding, you know, what you get from this very simple self-supervision and what biology, you know, these models capture. And, you know, the plot that basically shows that frontier and the dependence on MSA depth might lead you to conclude that these models can't generalize because you would look at the low-depth proteins. And of course, if you want to move outside the space of natural proteins, by definition, those proteins will have empty multiple sequence alignments. So that plot might lead you to conclude that these models are going to be useful in areas of protein space which are well understood, but they will not be useful as soon as we try to go beyond that. So we really wanted to look at this in a rigorous way. This is a picture of sequence space. There's a great review by Pasu Hong and David Baker that I think first kind of drew this picture. And the idea is that sequence space, of course, is exponentially astronomically large. And evolution, even evolution with its billions of years of opportunity to search the spaces, has really only been able to visit some relatively small part of it. But of course, many of the protein functions and structures that we may want to create would be in a space that evolution has not visited. And you might have to run evolution many, many different times for it to ever reach that space at all. And so The question is basically, can these models generalize outside of the natural protein families that they've been trained on? And so we're going to construct a very simple generative model here. And this model is just based on two things, a probability of sequence, which is given by the pseudolikelihood of the language model. And a probability of structure, which is just a linear projection basically to a pairwise distance distribution for each pair of positions in the protein. And so the model's gonna project out, you know, a potential to the structure and it's going to project out a probability of sequence. And then we can use Bayes' rule to, to generate proteins. So this is showing now what this looks like when you apply the model generatively. And what we're really doing here is, you know, alternatingly sampling a sequence conditioned on a structure, and then a structure conditioned on a sequence, and descending this energy. And when the model kind of reaches an energy well, at the bottom of that well is a protein and a protein sequence. And then we're using AlphaFold as an oracle basically. So we're not using any information from AlphaFold, but we're taking that sequence, we're passing it into AlphaFold to get a prediction of the structure. And so these models can really explore a wide, space of protein structures. This is just basically a plot of that structure space. I think it's constructed with TSNE. And there's a diversity of proteins that are generated, everything from all alpha to all beta to alpha and beta proteins. And then overlaid on the plot is the sequence identity to the nearest natural protein. And so You can see that these generations, they really cover the natural space, but they also go beyond that. And so there are proteins here which are really distinct from natural proteins. And so we tested these proteins experimentally in a collaboration with the Baker Lab. And we synthesized 129 of these generations kind of across this space. And then we look at two things. We look at solubility, and then we look at the presence of a well-folded monomeric species by size exclusion chromatography. So it's really basically, you know, we're looking for elution at the right molecular radius, and that is indicative of a well-folded monomeric species. And so those are what we are calling experimental successes is ones where we, see that. And so overlaid on this plot, you can see the proteins which were experimentally successful according to that criteria. And then this is just to show that it's really covering a variety of secondary structure compositions. But now getting to that key question, which I began this part of the talk with, the question of, can these models really generalize. This map is showing kind of the relation of these generated proteins to natural proteins. And so on the x-axis, for each generation, we basically search a large compilation of natural proteins for, for similar sequences. And so anything that's to the left of that vertical red line has less than 20% sequence identity from any natural protein. And anything that's kind of on, you know, on the zero point there is just, there was no significant match within a natural sequence database. Then the second thing that we did is we took the closest matching natural sequence, and we folded it using AlphaFold to predict its structure. And so that's shown on the y-axis. So the y-axis is looking at the TM score between the generated, the predicted structure for the generated protein and the predicted structure for the nearest match in natural proteins by sequence. And so anything that's below the 0.5 line is a protein where that, closest sequence match is also predicted to have a different fold from the generated protein. So the proteins that are in this lower left-hand quadrant are very different from natural proteins as far as we can tell and by those metrics. And what we find is that there is a high success rate here. And so 63% of these distant generations succeeded in this experimental test. And of course, you know, this will need to be confirmed with high-resolution crystallographic studies. But, you know, these— so the size exclusion test has been shown to strongly be linked to that. So one last thing to highlight here, because I think it's really interesting and kind of shows, I think, something that the language model is doing that's different from these kind of structure prediction models. So we can also use these models to design sequences for a target backbone. So we took a set of, and here we chose 4 different backbones that were created by de novo protein designers to be distinct. From natural protein structures. So these structures have significant meaningful differences from natural proteins. And we basically ask the model to design sequences for each of these backbones. And so we're comparing this really simple model where the structure prediction is literally this kind of linear projection to the distance distribution. And then we're comparing, and so, We're optimizing this jointly with the language model loss. So it's really the joint distribution here of sequence and structure. And then we're comparing this to inverting AlphaFold. So you can think about this as inverting ESM or inverting AlphaFold. And there's a big difference in the experimental success rates, as you can see here. So actually, most of the ESM designs for these backbones were successful, and very few of the AlphaFold designs were even soluble. And one thing that you can see here is that, you know, if you look at the designs that were created by the language model, that's the left column of this new plot. You look at the designs that were created by AlphaFold, that's the right column. And again, we're just using AlphaFold as an oracle to predict the structure here. And then we're overlaying AlphaFold's PLDDT at each position. So how confident is it in its prediction for each position? AlphaFold likes both of these designs. So it likes the language model designs, it understands them well. It likes the protein designs that were created by inverting AlphaFold as well. But when you look at the language model's understanding of the designs, what you see is that well, not surprisingly since we've been optimizing this quantity to generate the language model structures, the language model assigns a very low perplexity to these proteins. And so it has a good understanding of them. They look like coherent proteins to the language model. But when you put in the designs which have been inverted by, you know, produced by AlphaFold inversion into, the language model, what you see is very high perplexities. And so this is really saying that the language model doesn't think that these sequences are very likely. And so, you know, these probabilities are really distinguishing between the two sets of designs. And so, you know, one possibility here is really that the language model is kind of learning this global coherence of the protein, whereas these structure prediction models don't need to do that because they just take the sequence. They learn to complete some patterns, but they don't need to actually learn the coherence. Maybe it's worth saying one more thing here because it's really interesting, this question, because we see this dependence on the sequence depth. But then we also see this model can really go outside the space of natural protein. So how is it doing this? How is this possible? And I think one possibility and something that's kind of interesting to think about is maybe there is this deeper grammar that's learnable from evolution, a set of patterns or motifs that links sequence to structure. And the language model is able to learn this. And of course, what's interesting, the language model has never been trained on structures, really just trained on sequences, but it is sort of picking up that kind of latent variable, that hidden variable of the structure that's influencing the sequences that we observe across all of nature. And so, you know, maybe there is this evolutionary grammar that, you know, is an efficient means then to generate proteins that can really go beyond the structures. And sequences that the model, that nature has created and that the model has been trained on. Okay. So finally, I just want to show one last thing, which is you can really extend this idea and actually, and this is one of the things we're very interested in doing is, you know, can we make biology programmable the same way that we can program computer systems. And so can we write down basically a programming language and express a set of constraints on structure and function in a high-level programming language? And then can we use generative AI basically to find designs that will satisfy those constraints? And so this is now using ESMFold. And so we're really doing this at a high resolution. And so with ESMFold, we can get atomic level coordinates for any atom in the structure. And so you can really construct a language that allows you to specify atomic resolution details where you would like to have that and leave the overall architecture plan to be solved through much higher level abstract. Constraints. And so we're going to write down, you know, these constraints and these designs in a simple programming language. We're going to parse that into an abstract syntax tree. We're going to compile that into an energy function, and then we're going to— and each energy function then specifies a generative model that we can sample from, or I guess optimize via simulated annealing to produce designs. And so this is an example of scaffolding a constrained site. And so the idea basically here is the region in blue, we've specified the atomic level coordinates of each of the side chains in this binding site. And then we're asking the model to find a scaffold that holds, holds those two alpha helices in the correct orientation relative to one another, and also holds the side chains in the right places. And so, you know, this is just an example of what that type of design looks like. And then you can specify these higher-level constraints on the structure, such as symmetries, which are shown here. You can generate larger proteins and assemblies. So the left one has 450 amino acids. The second has 600 amino acids, and the last two have 720 amino acids. And all of these have these programmed symmetries. And you can design very complex symmetries. For example, a symmetric protein that are complex structures, symmetric protein that has 3 binding sites, for example, symmetrically arranged. Proteins that have multiple subunits with different internal symmetries. So yeah, I want to leave with just one final thought. And so, you know, I think we're seeing across generative AI and unsupervised learning, and this idea is coming increasingly to the forefront, this idea of training on the Library of Babylon, infinite library that when you, you know, in the limit of data basically, and with increasing model scale, as you compress that data with a more and more expressive model, you know, what's kind of emerging is this idea of a world model or basically a simulator because it's really that, you know, those are the constraints that are expressed in in the data that we're looking at. And so in biology, you know, I think what we can really think about these models as doing is learning emergent simulators of the data that they are modeling. And it's, you know, here we're with sequences, you know, then sequences of proteins, you know, it's really an emergent simulator of, you know, those underlying constraints on protein, their structure and their function. And we've shown that then you can really invert that simulator to design proteins. And so I think this is a very exciting idea as we continue to generate more and more data and as we continue to scale these models and we're going to extend this beyond proteins, of course, to model RNA and DNA and cells. And there's a lot of really exciting work happening right now to really look at unsupervised learning for all of these different modalities. But at the end of the day, you know, what we're creating is, is these simulators. And, you know, it's, it's, it's so different from the classical idea of a simulator where we really thought, you know, classically what, what we do is like write down the, the, the, the physical rules, the, the chemical principles. We kind of build this thing from the bottom up. And of course that so far has, has really proven intractable. So this is a very different way to think about a simulator as a, predictive model. But if it's invertible, you know, you can use it to understand and design biology. So that's really what we're working towards. And I just want to highlight this is the work of a large team, the ESM team at Meta AI FAIR, a group of whom have now left Meta to create a new company. And then The work that I showed on experimentally validating these generated proteins was in collaboration with Basil Wikki and Lucas Milaes in the Baker lab, and Sergei Avchenikov and Justus Dauperès. So I'll end there. I have a couple of minutes, I think, to take questions. Okay. I'm going to run around with these if there's any other questions. Yeah. So any questions? Yeah. Yeah. I was just running, wondering with these generated proteins, did you get any sense how they would actually function in a living cell? So we've not looked at function yet. You know, we really wanted to start with structure because we can look rigorously kind of at this question of generalization. But of course, function will be the next step. I think it's really cool stuff. I'm curious about if you could estimate, like, if someone just wanted to do this on AWS, how much it would cost to train one of your models and then maybe speculate on what that implies about sort of the, you know, upper 1% and lower 99% who can make models like this to advance science. Yeah. Well, it's a good question. I don't know what it costs to train ESM-2 now, but it's gone down a lot since when we trained it. So, I mean, yeah. And it also sort of depends on the, depending on the cost structure of where you train it. But yeah, I think it's still probably pretty expensive to make ESM-2, probably. I don't know, it might be less than a million now. I'm not sure. But really for a large-scale project, compared to trying to build CERN or something like that, this is really cheap. So it's something to think about. But yeah, as we want to scale up to hundreds of billions of parameters or trillions of parameters eventually, which is of course what we want to do, you know, that would cost hundreds of millions to train models of that scale. So it is very expensive to create these models. And I mean, it's, you know, I think this is one reason why open source is really important. The models we've created so far have been open source, and so people can build on them. You don't have to go and train them again from scratch. The interesting thing about these improvements in compute too is just that, you know, from a time standpoint, you know, it's moving so fast that it becomes increasingly easy to train the frontier model. So it'll be interesting to see what happens. I'm very curious about the deeper grammar that you speculated about. So the language model is designing proteins that are different in sequence and different in structure from natural proteins, but we hypothesize that there is something that they have in common. So how do we decipher what that deeper grammar is and what it means from a physical point of view? Yeah, I mean, it's a really interesting question. It's mysterious. I mean, there has been some work on this. I mean, one, So Ghevor Grigorian's group has several papers on the idea of terms, which are kind of these motifs which link sequence to structure and have been used across evolution. And so, you know, maybe just because for given structures, there are certain sequences that are optimal, right? For each of those parts. So I think there's like kind of two things you have to do here, right? Because you have to get the local, you have to get the local geometry right. And typically there's kind of multiple sequences that would be optimal for each kind of part of local geometry. And you have this kind of global coherence problem where you have to then I think like assemble each of those local motifs in a way that, you know, leads leads to the overall fold being realized. So you need to assemble them in some kind of coherent or compatible way. And out of the set of choices that are possible in every position, pick the ones that are consistent with kind of a structure— well, a sequence where its energy minimum through folding is kind of the structure that you're trying to design. So it's a super hard problem. And I don't think it can be solved just by picking the locally optimal motif at each position. And so I think the model kind of has to do two things. It has to understand that locally optimal set of assignments, but then it has to understand how to kind of put that together coherently. So I think that was something like that. More of a high-level question, but even though the languages are completely different, like natural languages and protein language and DNA, for example, we keep using the same training task like masked language models or auto-regressive models. So does this imply that the room for improvement for injecting more inductive bias or domain expertise shifts more from training to post-training or model evaluation or downstream tasks? Yeah. Or do you think there's still room for improvement during training as well? Yeah. So it's, you know, there's this frontier with compute, just like compute efficiency front here, right? Because like it's very costly to include data or objectives in the pre-training, which is why I think the Transformer model has dominated, why these simple objectives have dominated, because you can have this kind of two-phase training where you first train the base model and then you fine-tune it on a specific task with more data. But I, I do think that the field is moving toward also incorporating more into this training. So I think that's a really interesting overall direction is how can we supervise the right kind of world model or simulator and what data do we need in pre-training and what architectures do we need in pre-training to most efficiently do that per unit compute or per unit data. Like you have the sample efficiency with data and you have kind of the compute efficiency and you have a frontier with both of those things. Great. I'm conscious of time. So I think we should wrap up there. Let's thank Alex again for a fantastic talk. Um.