Scientist Stories: Danny Diaz, Engineering Proteins with 3D Convolutional Neural Networks

Awesome, let's get started. Hi everyone, thanks for coming. It's my pleasure today to introduce our speaker, Danny Diaz. Danny is a fourth-year graduate student in the Elkington Lab and a Machine Learning Lab Fellow at UT Austin. He has a background in synthetic organic chemistry and enzymology prior to pursuing a PhD in AI protein engineering. He is the creator of mutecompute.com, which is a machine learning as a web service application that brings structure-based deep learning algorithms to the academic community. And he's also recently founded a startup called Intelligent Proteins LLC, where he is commercializing the machine learning platform he developed throughout his PhD work. So please join me in welcoming Danny, and Danny, it's all yours, take it away. Hi everyone, thank you for being here. I guess I'll start sharing my screen. Um, can y'all see my screen? Yes. Um, yeah, so I'd like to thank Kevin and Jodi for inviting me today to speak and to present my research. Recently, we published a paper on plastic eating proteins, but I decided to make the talk talk a little bit more than just the plastic-eating protein, because it's kind of hard to talk about one protein for 45 minutes. So that being said, today I'm going to talk to you guys about engineering proteins with 3D CNNs. So before I begin, UT requires me to disclose that I started a company based on my research with my co-founder James Loy. So let's get started. So first I'm going to give some background on why we would like to engineer proteins. Then I'm going to talk about MuCompute, what it is and how we've used it to engineer a few proteins. And then from there, I'm going to talk about MuComputeX, which has been most of my, from a research perspective, most of my PhD work to expand MuCompute's capabilities to 3D CNNs. I mean, to expand it to new types of chemistries. So, most people know that proteins are just primary sequences of amino acids, so just a string of amino acids, and it's how these strings fold into 3D shapes that endow a protein with its function. So here's an example of a few proteins I found pretty interesting. On the left, we have some transcription factors, It's a homodimer, and given the sequence, the amino acid sequence, this protein is able to bind DNA. Here in the middle is a membrane protein that acts like a pore. You can see clearly that it's a pore because things can flow through the middle, and all of this is encoded in the primary sequence of that protein. And on the right here, we got an ATP synthase, kind of is the reason why all of us breathe oxygen, right, to like power this protein. It actually spins to generate ATP. So the main point here is that it's all in the 3D shape that a protein has its function. And then just to hammer this point home a little bit, here is a protein and a unique 3D shape that kind of impacted all of our lives these last 2 years. And this is the SARS-CoV-2 spike protein. And what happened was it made some mutations on the receptor binding motif, which and endowed it with the function of binding the human ACE2 receptor, and it was able to jump organisms and began the pandemic that we currently experience. So essentially, proteins are at the core of biotechnology, and the ability to encode specific functions into the DNA is what biotechnology is all about. And as we keep doing research and learning on how to encode a desired function into a DNA sequence, we will enable more cancer therapeutics, plastic-eating enzymes, so many applications, including vaccines and biomanufacturing for chemical commodities. The way I like to see it is as we learn to encode information into DNA, everything that biology can do would become engineerable. So transitioning now to how we train convolutional neural networks on proteins, we essentially treat 3D structures of proteins as pictures or images. So usually when you think of an image, you think of a JPEG. Here's a picture of a bird on the left, and usually these birds— these images have 3 channels: red, green, and blue. And they're usually 2D, so x and y. Well, for proteins, we'll have 3D images, so we'll have x, y, and z. But instead of having red, green, and blue channels, we have atom channels, charge channels, solvent accessibility channels, and whatever other physical channels we'd like to have. And this is kind of like the analogy between a regular CNN that's on your iPhone that puts a box around your mom's face to like how to the analogy to proteins. And our dataset usually consists of the Protein Data Bank. This slide's a little old. Whenever I made it, it was, uh, had only 160,000 protein structures. I checked yesterday and we broke 190,000. So the database is expanding really fast. But even more importantly, the scientific breakthrough of last year was AlphaFold. And that enables us to generate, to fold proteins computationally and drastically expand our dataset for structure-based deep learning. So the kind of models we use, they do, we do self-supervised learning. So it's completely unsupervised, there's no experimental data. And this is actually what enables us to apply these models to any proteins really fast because we don't require any experimental data. So the learning task that we do is we select a residue, an arbitrary residue, out of protein. So you can see we selected this asparagine shown here in pink. Once we've selected a residue, we have a coordinate in space. We usually do the alpha carbon, and then we can then filter out all the atoms within a certain distance from this alpha carbon. So you can see that here in the middle. So we call this a microenvironment. And then we generate a learning task by deleting the residue we selected. So we delete all the atoms in the asparagine, giving us the masked microenvironment. And now because we know what residue we masked, we have an artificial label that's generated. And we like to think of this kind of as having evolution provide the learning signal during training, because all the proteins that we're using are from nature, so they've had millions, hundreds of millions of years to evolve and mutate and to give us these stable 3D structures which we can observe and crystallize. So after we generate mass microenvironments, we have to kind of format it or digitize it for these CNN models. So here is the original digitization for the MuCompute model. We essentially had a 7-channel representation with about a 20 angstrom cube. So we had elemental channels shown here: carbon, oxygen, nitrogen, sulfur, hydrogen, and then partial charge and solvent-accessible surface area. And we do this for every mass microenvironment in our training set, and whenever we run any predictions on a protein. So kind of putting it all together, We take mass microenvironments, we pass them through a 3D CNN. The architecture is in the paper we published in 2020. And then at the output of this model, we, we do a softmax to see how probable each of the 20 amino acids are. So this last vector here is 20 numbers, and it's the probability of each amino acid belonging in the masked microenvironment. So we originally got this training scheme from a paper by Torn and Altman, and the grad student before me worked on it whenever I was a first-year student. He brought it up to about 70— about 70% accuracy at re-predicting the wild-type amino acid or the amino acid that we masked. And at the time we were looking at, well, how do we use this for protein engineering? And we came on this hypothesis of like, is, is the model actually making a mistake, or is nature making a mistake? So essentially, whenever the model does not predict wild type, is it maybe because wild type shouldn't be there? Maybe, maybe evolution hasn't gotten to the point yet to evolve this residue. So kind of diving deeper into that, here's different routes, different types of predictions the model makes or mispredictions the model will make, and this is kind of how we use them for protein engineering. So for example, here is a BFP protein we engineered in our 2020 paper. You can see at valine 216, the model tends to evenly distribute probability across multiple amino acids. This residue is probably very promiscuous and can allow a lot of different amino acids and types of chemistry here, which could be useful for engineering, but it's usually not the first types of mispredictions we prioritize. Another type of mispredictions are, for example, here at tyrosine 14. You can see the model is pretty confident that it should be a phenylalanine instead of a tyrosine. But if you go look at the probability distribution, you can see that the model is pretty confident that it should be an aromatic here. So the model knows that an aromatic should— belongs here, and it would prefer a phenylalanine, but we already have tyrosine there, so like the chemistry isn't changing significantly. The ones that we'd like to prioritize for engineering campaigns are like— are mutations like at glutamine 39. So for example, glutamine is a polar neutral amino acid, but if you look at the probability distribution, arginine, a cationic amino acid, got 68% of the probability, and lysine, another cationic amino acid, got another 30% of the probability. So the two primary cationic amino acids got 98% of the probability that belongs in this microenvironment. And then the third amino acid got histidine, which can also be protonated and be cationic, got 1%. So 99% of the probability at this microenvironment was assigned to a cationic amino acid. These tend to be the ones that we prioritize for engineering because there's a fundamental change in the chemistry that we're putting in the mass microenvironment. We're moving from a polar neutral amino acid to a cationic amino acid. So just to like make this analysis a bit easier, I worked with my college roommate over the pandemic to build like a web GUI for this, available at newcompute.com. We call it MuCompute View, and it's just an interactive visualization where you can like click on the protein, spin it around, select on the residue. It will show you the atoms, and you can see what's going on in like the microenvironment at the that the model is using to make its predictions, and then it will give you all the numbers in the top right-hand corner, so you can like inspect your protein and make your designs all from the web. So whenever we first made this model, we, we were working with Ross Thayer. He's now a professor at Rice. He wanted to validate this on 3 different proteins. I'm not going to talk about these 3 proteins in this talk, but you can go see all the data in our ACS in Bio paper. But we first wanted to see if what the model is mispredicting, how well does that correlate or overlap with other computational tools such as Rosetta and FoldX. And you can see the mispredictions from our 3D CNNs were very different from Rosetta and FoldX. And this is very interesting to us because it's like a new source of orthogonal information for protein engineering., and all the ones in red were whenever we actually experimentally mutated these proteins, we found another amino acid that improved protein function. So, moving away from these 3 proteins, I'm going to transition over to talking about the recent protein we engineered that actually got me invited to give this talk. So, we recently engineered a PET-eating protein, and the reason why this is such a big deal is because plastic degradation is a big problem. You know, plastic is so amazing because it's so durable, but its durability actually is like being— it's very bad for our planet. You know, we produce millions of tons a year and they enter the ocean. They're just accumulating in the environment. So we had been working on for a while to engineer this PETase And this protein essentially takes polymers of terephthalic acid and ethylene glycol, right? So here's the polymer PET, polyethylene terephthalic acid, and it will degrade it down to the two, the monomers. And we also see some monohydroxyethylene terephthalate and dihydroxyethylene terephthalate. And so we just ran MuCompute on the PADDase structure and took— just ranked the designs by low wild-type probability and just took the top mispredictions, again prioritizing the ones where the chemistry fundamentally changes, and we saw massive improvement in protein stability and protein function. So here is a— we named our variant FastPADDase, and all the engineering— most of the engineering work was led by Hong Lu, And, uh, yeah, so here's us comparing it to other PET depolymerizing enzymes. So some of them are PETase scaffolds, such as this Thermopetase and DuroPETase. Other ones are cutinase scaffolds, like this LCC and ICCM. And you can see that by using new compute designs, we were able to significantly outperform the literature on the, on PET degradation. And then we have— yes, we have two questions in the chat. Perfect. Yeah, Bryce, do you want to ask your question? Sure. My question was just, are you doing, um, self-supervision, um, only on the target protein of interest, or are you doing it on like a diverse set of proteins, some pre-training, um, from UniProt before you consider, say, your target protein could be like this FASTPase Yeah, that's, that's a good point. I kind of glossed over that a little bit. So our training set was 1.7 million microenvironments across 19,000 protein structures, and these proteins were selected with a max sequence similarity of 50%. So that's how we kind of made sure all the proteins are very diverse. So no two proteins in the dataset were over 50% sequence similar, and then we would subsample those proteins residues to generate their microenvironments. Cool. So let's just miss that. Yeah, sorry, I should have said something about that. Yeah, so this is really— what, this is really powerful because we don't require any phenotype data or any like specific protein training. We just train across all proteins and then we can just quickly run inference on any protein structure. I think there was a second question. Yeah, Ben, you want to read the question? Yeah, I can do that. Yeah, my question was basically just why— I didn't quite get how there's a connection between the protein function and what you predict, which is really like the spatial— like what makes sense of what residue should go there. How do you— why do you make the relation between protein function and the model predicting that that's the best residue that should go in that masked area? Yeah, so, um, that's a very good point. So we are not training at all on protein function, strictly protein structure. So you can kind of think of it as like we're training the model to look at proteins that have evolved over millions of years to be— form these stable folds to the point where they can crystallize and be put into a database. And we're training the model to predict each amino acid's— what local chemistry does each amino acid like? What are all the different modalities that you would expect to see each amino acid in from a structural perspective? And then based on that, we have just empirically observed that if you just make the protein, I guess, more happy or remove amino acids that are incongruent, with their surrounding protein chemistry that overall leads to improved protein phenotypes. And phenotypes is very multi-dimensional, right? One phenotype is expression ability, thermal stability, function is just one phenotype, and different mutations manifest themselves in different phenotypes. It just happens that the mutations that we screened here— we, so we make a lot of mutations that the model predicts, and then we just screen for the phenotypes of interest that we care about. So like a lot of mutations that don't make it into the final variant does not necessarily mean they don't improve phenotypes. They just didn't— they might— it's very possible they don't improve any phenotypes, and we have observed that, but we just try to select the ones that improve the phenotype that we're interested in, which is usually activity and thermal stability, and in this case also pH stability. Does that kind of answer your question? Yeah, it definitely does. It makes sense. Yeah, it makes sense evolutionary-wise. And yeah, that's fascinating. Thank you. A quick follow-up question, please. How do you determine what mutations lead to specific phenotype changes? Are you doing that computationally, or you're taking all of the variants and screening them experimentally? Yeah, so it's all done experimental. We usually just grab the top 10 to 20, maybe 30 mispredictions, and then just make those experimentally. And then we usually get a good enough hit rate where there's a handful there that look promising for the phenotypes of interest. And then we spend the rest of the time just figuring out how to combine them. Is there any more questions before I move on? These are great questions, by the way. Okay, so, um, yeah, so we did, as I just stated, made, you know, a handful of— a good amount of variants here, screened them for thermal stability and activity and pH stability, stacked the ones that look most promising together, and, um, that's how we made FAST-PETase. And then here it's just kind of diving more into like the the qualities or the— So you can see on the left, for— so H-E-B is just a grocery chain here in Texas. We just went and bought some food products there, took the PET products hole punched out a disc, and you can see this one was about 1.5 days, this is about 4.5 days, and you can see it briefly correlates with the mass of the disc. So you can see over here, this mat— this disc was about almost 11 milligrams, this disc over here is 7[No speech] We're able to get 100% degradation at 40°C, so that's like a hot day in Texas. But that took longer than 2 days. I think that was like 4 to 5 days, so still pretty impressive that we can go from plastic degrading over hundreds of years into a matter of few days to like a week or two. So, um, the next protein I'm going to show you guys, uh, has to do with cancer. So this is another project that we haven't published it yet, but, uh, essentially when, uh, in cancer Uh, they, they tend to— cancer cells tend to reprogram metabolism, and they tend to, uh, because the goal of the cell now isn't to be healthy, but it's to reproduce and metastasize, so it needs more resources. And so it would prefer to like consume the resources externally, so like steal it from the extracellular matrix rather than produce them themselves. And turns out that, uh, serine is the third most consumed nutrient in cancer cells behind glucose and glutamine. So serine is just one of the 20 key amino acids, and that makes sense, right? Because this— the cancer is trying to metastasize and grow really fast, so it needs to make a lot of proteins. So you would much rather, you know, take in amino acids to make these proteins rather than biosynthesize it themselves. So, uh, we've been working on a project down the hall here in the Giorgio lab to engineer serine dehydratase. They've been working on it for a while. Ebru Kaya was the PhD student who was leading this project. So this protein essentially takes L-serine, uses vitamin B6, and it produces pyruvate and ammonia, essentially degrading serine. And they have been working on this for years, and they were primarily using error-prone PCR to find hits of mutations that would improve activity of the enzyme. So you can see this is where they started. They've been accumulating mutations through error-prone and improved the activity with serine and threonine as the substrate, but they still lacked— the protein still had terrible expression. You can see that down here. So they would get like about 8 to 10 milligrams per liter when purifying the enzyme. And so we collaborated with them, gave them some designs,, and one of the designs significantly improved the expression ability of this protein, right? So this is a different phenotype. Again, we didn't train the model to improve protein expression, we just trained the model to help make protein chemistry more congruent, and this is one of the byproducts. So they cared about expression ability, that's what they were screening for, and then that's how we filtered out which design to keep. And you can see we got a 7, 8x, and improvements in expression ability of this protein, and actually got a small improvement in activity as well. There's two more questions in the chat. Sure. Also, um, your question. Hi, uh, so I was just asking about, um, I guess dealing with, uh, with rotational bias perhaps in the training of your, of your model. And so is there any, is there any way that you're dealing with that? Yeah, these are great questions. I'm usually, I'm not used to talking to the people who are more like the machine learning crowd. It's usually I'm talking to protein engineers. So the way that we deal with that is whenever we center on a residue, we usually use the backbone orientation to make sure our boxes are oriented the same every time. So we'll like take the alpha carbon nitrogen bond as our like x-axis, and then the alpha carbon carbonyl bond as like our y-axis, and then we'll take the dot product to get the z— I mean the cross product to get the z-axis, and then we'll orient it where it's pointing towards the R group is in the positive z. And then we do that for all amino acids to make sure that the side chain is always in the positive z. Does that make sense? Yeah, that makes sense. And I guess then when you're masking those positions in your target proteins, you're doing the same thing. You're aligning them along kind of that— the backbone. Yeah, yeah. So we orient the box where the backbone is always pointing in the same way, and the side chain that we mask should be in the positive z direction. So we do that in training and in inference. Okay, thank you. I think there was one more question. Yeah, hey guys, I have a very large time zone difference. It's night in here, so I'm trying to be quiet. I'm more familiar at this kind of techniques applied to large language models trained on huge corpora of protein sequences. And I was just wondering, do you think the same technique with mispredictions would work for this kind of models because they don't need structural information to be trained on, which I think like it helps to train model on larger corpora of protein sequences. Yeah, so there are language models that do this. This is essentially like masked language modeling where you just like mask an amino, a residue as a token. I think the ones that I'm most familiar with is like ESM-1B and ESM-1V out of Facebook, where they can do similar— where they'll output, I guess, like a probability of a sequence, and then you can just change a token to whatever amino acid you want there and then get the probability of that sequence. So very similar to the output of these models, which I would actually— I actually have some benchmarking results toward the end where I will compare some of the new compute models to Facebook's ESM-1B model. Is there any other questions? Similar to the language models where they would train on a large corpus of amino acids or sequences like UniRef50, and I think there have been improvements when you take those models and find train them on, say, a family of homologous proteins. Have you investigated that with MuteCompute to see if narrowing down to homologous structures or the amino acid environment from homologous structures improves the performance? So we are working on transfer learning it to different protein families, but it's a little too preliminary to kind of discuss those results. But the difference that I would say is whenever we've done some studies on comparing sequence-based models to structure-based, so these new compute structure-based models, and what we've noticed is the structure-based models tend to generalize very well. We tend to get about very similar misprediction accuracy irrespective of the protein fold or protein sequence. While we tend to see the language models tend to perform really well on certain types of sequences and perform very poorly on other types of sequences, which I think is a relationship to how similar sequences are in the dataset. While our models don't have that many structures in the dataset, but they tend to generalize across most structures. I don't have data here to show that here, but we have some publications coming out soon that will show that. So, um, I'm going to move on to the next protein unless there's another question. Okay, so, um, this is another protein where we just published this last year in 2021. This is a BST-LF polymerase. So this is used for LAMP assays. So LAMP is essentially like PCR but at one temperature, so you don't have to cycle the temperature. It's isothermal, and our Dr. Ellington, my professor, really wanted to engineer some LAMP polymerases, especially because when the pandemic hit, we were trying to like target regions where, you know, a PCR machine or like a lot of biotechnology infrastructure was not available around the world. You can think of like South America, Africa, and all these other places, But like, you know, having access to equipment that can just hold the steady temperature is more readily available. So here is this polymerase 3 engineer again looking at just the top 10 mispredictions. So you can see here is the probability assigned to the wild-type amino acid, and here's the probability assigned to the— this is the— to the amino acid that had the highest probability. So you can see here was leucine, got 74%. Asparagine got 85%. So we took the top 10, made them all, and then worked on combining them to see, to improve the protein's function. And these are the results we got. So ideally you'd like to run the assay at a hotter temperature so you can get results faster, right? Nobody wants to wait an hour or 2 hours for their COVID PCR. We'd much prefer to wait 5, 10, maybe 15 minutes to get a positive or negative result. So this BR512, I think, is already a commercially available LAMP polymerase. So that's the dashed line. And then this is our starting point from engineering, is the orange line. And then the blue line is the triple mutant variant that we made after combining the top hits. So you can see at 65°C, which is usually where most LAMP assays are done at, we got significant improvement from orange to the blue line. So we were able to get a result from— it moved from about 24-25 minutes down to about 12-15 minutes, and it was actually doing better than the commercially available enzyme. And then upon heating it up to 72, you can see our starting point that we started engineering from dies, and but the commercially available protein is still working. But we're getting now results in under 10 minutes at 72. And then when we heat it up to 73, you can see the commercially available protein also dies, and only our engineered variant still works at 73, again getting diagnostic results in under 10 minutes at a single temperature. And then once we got to 74, all proteins died. I think we're still further engineering this protein. I'm not sure. But this is just really cool to show how these structured-based models can give us designs that can quickly improve phenotypes in an unsupervised fashion. So now that I kind of convinced you guys that these structured-based models can be used to engineer proteins, I'm going to kind of shift gears and talk about uComputeX. And this is kind of a lot of the data engineering, machine learning stuff that I've been working on throughout my PhD. So NuComputeX is essentially us trying to expand NuCompute's capabilities beyond amino acids. Due to computational data engineering reasons, the model was only able to see amino acid atoms. It was unable to see non-amino acid atoms. So you can imagine at a protein nucleotide interface, like at the protein polymerase I just showed you, all those mass microenvironments, they weren't able to see all the atoms that belong to the DNA or RNA or any nucleotides that were there. Similarly, for like, you can imagine like a protein active site, like for the serine dehydratase, or for the PETase, the model is unable to see any cofactors that are present, any metals that are present, any ligands, any glycans,, and also for like more like vaccine antibody therapeutic types of applications, these protein sets will also be very glycosylated. So like the model wasn't able to see any of those sugar molecules like you would see like in an antigen antibody interaction. So to drive this point home and kind of show that, you can see here for nucleotides, this is kind of what MuCompute was seeing for this arginine. So this, this here shown as orange spheres is the mass residue that's been deleted. And this is what the microenvironment would look like on MuCompute, and this is what the mass microenvironment would look like with MuComputeX. Similarly for ligands, this is a transcription factor that is activated upon binding a ligand. The ligand is shown here in blue. So you can see for this histidine residue shown in the middle for MuCompute as orange spheres, When it's masked, this is what the environment would look like from U-Compute, and this is what the environment would look like from U-ComputeX. So now the predictions that we get out would take into account the ligand that we put there. And that ligand could be present either from a crystal structure, or that ligand could be present due to computational docking. So this is a good opportunity to kind of point out where a masked language model, a transformer-type model on sequence data, would not be able— you would not be able to pass this information, this DNA or ligand data, to those models. So to kind of make this happen, we had to redo our entire data engineering stack, and I'm kind of going to dive into that. So this is the data engineering stack we built. So this MuCompute 2 one is the web application that's running on mucompute.com right now. It's what I kind of talked about where you can interactively visualize the mutations. And then we, we built these 3 data engineering libraries. So we built apchem, so this is responsible for getting all the chemical properties and all the structures looking correct and looking uniform so we could properly digitize everything in apbio. And then apbioinformatics is where we go, and it's how we generate different datasets to maybe like bias or transfer learn or like sample these protein-ligand and protein-DNA interfaces. So diving a little deeper into APChem, what really kind of made this possible was that we moved away from using PDB2PQR to get our partial charges. So PDB2PQR gives you a partial charge for every atom by using force field lookups, and then you would have to provide your own partial charges for non-protein protein atoms. So this made it really hard to kind of scale because we wanted partial charges that would take into account like how a ligand polarizes the protein when they interact, or how a protein would polarize a ligand when they interact, not have them be calculated separately, or they're not even calculated, they're just looked up in a table separately and then just put them together. We instead use a this physical chemistry method called the electronegativity equalization, and then this will— it's molecule agnostic, so it doesn't care if the atom is from serine or threonine or glutamine or some amino acid. It just kind of looks at the atom and looks at the local chemistry going on, the local geometry of how the atoms are oriented in space and the atoms that are nearby to determine a charge on each atom. So that really kind of bridges the gap Whenever, you know, you have some carbonyl oxygen that is hydrogen bonding with backbone amides of a protein, right, and those backbone amides are polarizing that oxygen, so you can make an oxyanion hole, which is what you typically see in like hydrolases, right? So it allows us to get that minutiae level of detail correct, so the model can learn these, can really get more accurate predictions. Additionally, we moved away from using a PDB file format to a CIF file format, which allowed us to put all the information into one file. And that is what enables us to stream the data, which I'll talk about in an upcoming slide. So before we— One more question in the chat. Sure. Calvin, you want to ask a question? Sure. Yeah, so I was just wondering if there are particular types of phenotypes where you think this approach would not struggle to optimize? I'm curious if you have intuition around that. So you're asking me what phenotypes you think the model would not do well on? Yeah. If I would guess, it probably would be phenotypes that are strongly dependent on disordered proteins, or whenever, like, upon a ligand binding, a disordered loop becomes structured or ordered, because our approach is strongly dependent on having an accurate input structure. So a lot of these disordered proteins, the input, if you have a disordered protein, if you do get a structure, which I doubt you could, it's just probably not very accurate, so it won't translate from the computer to the bench. Other phenotypes I would say, which I'm kind of trying to get experimental data for through collaborations right now, but my hunch would be that we probably aren't too good at it, And that would be predicting mutations to make proteins adapt to cold temperatures, just because most mispredictions that we make, right, if you make it more congruent, it stabilizes the protein, which means it'll become more stable at hotter temperatures. So how do you kind of design proteins to go down in thermal stability in a way that's like functionally active? I haven't really thought of an intuitive way to make designs for that, and I don't think the model really could help there, but I don't have any data yet showing that it cannot either. But off the top of my head, those are the two that I can think of. Okay, so yeah, so this— these changes here in apchem are really what enable us to make new compute X happen. It's what enables us to be able to not have a corrupt microenvironment for non-protein atoms, where we can have uniform representation of all the chemistry, so we can train on protein-X interactions. So now that we can kind of digit correctly, uniformly represent all the data for all atoms that we would see in a protein structure, we had to expand our microenvironment to have a phosphorus and halogen channel. We're currently working on implementing a cation channel. But we're still playing with how to get the partial charges correctly on cations, like, so for, like, transition metals, like, how would we differentiate, like, a nickel 2 versus a nickel 1, or a copper 1, copper 2. But so here we have 2 new channels, and now the partial charge and solvent accessibility channel also have values for the non-protein atoms. So for example, you can see here in this mass microenvironment, we masked a serine residue shown here as green spheres. This difluorocyclic GAMP, you can see here in purple, we have two phosphorus atoms, and in orange right here and right here, we have two fluorine atoms, and that's what you're seeing here, right? So here's one fluorine atom, here's the other, and here are the two phosphorus atoms, and you can see this yellow dot down here is this one sulfur atom. So now we can properly represent the entire chemistry and show it to the model. So the next thing is we have to kind of customize our datasets now to make sure we're sampling the chemistry where these phosphorus and halogen and ligands and DNA molecules are. So the approach we've been taking to do this, uh, we've made datasets where we cluster sequences at different sequence similarities, so 50, 60, 70, 80, 90, And again, these are sequences of protein structures, and this gives us about 23,000 to 35,000 structures. And then once we have all these datasets, we then go to these structures, and we have an AP Bioinformatics. We have all these filters that will pull out residues that are at protein-protein interfaces, protein-ligand interfaces, protein-carbohydrate, protein-DNA, RNA, metal interfaces. And then once we sample all of these interfaces and pull out all the residues that are interacting with these, uh, at all these functional interfaces, we then will backfill with random sampling to make sure we sample each protein sufficiently. And then, so this is kind of the amino acid distribution in the 50% sequence similarity data set upon doing this, and this is kind of the distribution that we observe. Just by taking the proteins based on these sequence similarities and doing this— applying this algorithm to it. So for the 50% sequence similarity training set, we had a 90/10 split, giving us about 2.3 million residues in the training set and then 2.6 million residues in the test set. So to train these models, we built this library APBio, I kind of showed you the output of APBio. This is— this 4D tensor is the output of APBio. We wrote it in C++. Here's the architecture of the software library where it essentially reads in your dataset as metadata. So for example, here is the dataset. You just kind of tell it what protein to go to, what chain in the protein file, what residue to center on, and what amino acid is there. And then you just— this allows us to quickly make datasets of millions of rows and store them as text files that are just megabytes instead of storing millions of 4D tensors that are terabytes. And it allows us to quickly iterate, smooth, change the datasets, and also change the representation that we put into the 4D tensor. And what we do is we just make all the data on the fly where you read in one of these text files and you just point it to a directory full of protein structures or CIF files and it just populates a queue on RAM of microenvironment 4D tensors. And then the, the model which is on the CPU, on the GPUs, just has to ask for the next batch. So this allows us to do just-in-time data generation during model training. So you can see here in orange is the time it takes us to generate a batch And then in blue is the time it takes us to backpropagate on that batch. This is on one GPU with 100 boxes, so 100 microenvironments per batch. And then we use 48 threads for the data generation. So those are all the data engineering changes we've made. And then we've also moved away from a traditional CNN to a residual convolutional neural network. And so to kind of come up with a starting architecture for this, we use Ray Tune to span the hyperparameter space, play with the number of neurons at different layers, play with the dropouts and the dense layers. We just use Bayesian optimization with Ray Tune, and we usually save the models iteratively while we're training as they hit different training accuracy checkpoints. So training models for the 50, 60, 70, 80, and 90% sequence similarity while saving them as they hit different accuracy checkpoints, that gave us over 6,300 models that we saved. So then we— here's the performance of the top performing models from this. So you can see the original MuCompute model got to about 69% accuracy. And here's the performance of the best 50%, 60%, 70%, 80%, and 90% sequence similarity models. You can see they all hit about 83 to 85% accuracy, and they're able to learn relatively well all these different interfaces, with DNA being the worst predicted interface. So this learning task here, the numbers you're seeing is the ability to re-predict wild-type accuracy. So if we delete a residue, how well do you re-predict what we masked? That's very different from the goal of protein engineering, where it's not to re-predict the wild-type amino acid, but to predict the amino acid that would improve the protein, I guess, chemical congruency, which can then manifest itself over several different phenotypes. So to kind of gauge when do we start overfitting to the wild-type learning task and start moving away from the actual task of protein engineering. We benchmarked against FireProtDB. So FireProtDB is a database put together of just all single point mutations. So this is all experimental data. They have about 226 proteins that have structures where they have experimental data for point mutations, where they can, upon making a mutation, they measure how the change in temperature, so the DTM, so does the protein become more thermostable up or down, or less thermostable, and also delta delta G, so overall stability of the protein. So different experimental mutations have both or one or the other, and you can kind of see those counts down here that we did. So just taking the top performing ResNets and it's for 50% and 90% sequence similarity, You can see them here when we compare them against Pearson and Spearman correlations for delta TM and delta delta G against ESM1V and Rosetta's Cartesian delta delta G. You can see the ResNets outperform both Pearson and Spearman correlation for these against ESM1V. And Cartesian delta delta gene, and these are the number of mutations that went into these correlations. But correlation coefficients can be very misleading, and it doesn't really mean much when you're talking to a protein engineer. So here's another representation of the data. So here is now looking at— if you go to FireProtDB, And we look at all the experimental data where the mutation made the protein at least 2 degrees more stable, we have about 245 of those in FireProtDB. And if the mutation actually made the protein more stable by 1 kilocal per mole, we have about 200 of those in FireProtDB. And that's what you're seeing here on the left-hand side, and you can see Given that the mutation improved the protein by at least 1 kilocal per mole or 2 degrees Celsius, the RASnets are able to predict to make that mutation almost 50% of the time, between 40 to 50% depending on the model, but about 46-47%. And for delta delta G, it was about 45 to 60% with about 50% being the most common. Now compare that to ESM1B. So this is the language model that was getting to about 6.5% at predicting these mutations. And keep in mind that these ResNets are only 2.5 to 20 million parameters compared to ESM1B having 650 million parameters. And these ResNets are trained with 23,000 to 35,000 protein sequences compared to ESM1-B being trained with 98 million sequences. So that's kind of surprising there. And then on the other end of the stick, where if you make the mutation, the protein becomes less stable, either thermally less stable, so the temperature drops by at least 2 degrees, or the delta delta G becomes worse or increases by at least 1 kilocal per mole, you can see that the ResNets correctly predict to not make this mutation 95 to 97% of the time, compared to ESM1V being at 99 and Rosetta being at 91-94% of the time. So next, I'm going to kind of transition now into different applications that we're doing with MuComputex. So these are current proteins that we're engineering where I don't have experimental data yet to disclose. So essentially we're trying to engineer these biosensors, they're transcription factors. These proteins are essentially bind a ligand and then undergo a conformational change to control gene expression. We have protein engineers in our lab that do directed evolution to engineer this, so you can see here in blue is an experimental structure of a protein. We evolved it to bind new ligands, so this protein has about accumulated 10 mutations via several rounds of directed evolution, and you can see that we're able to AlphaFold this and move 10 mutations in sequence space pretty well, getting a TM score of 92. And we did this for 2 different proteins that we started with the same protein, accumulated 10 different mutations, uh, in 5 different ways to bind 5 different ligands. You can see here 2 of those structures that were evolved to bind 2 very different ligands. So you can see here in blue, this is a co-crystal structure, so this is experimental data of the evolved protein with its new cognate ligand, and you can see that we can alpha-fold the structure, and then dock the new cognate ligand relatively accurately. So then now we can use MuComputeX on these computationally generated structures to help guide the library design for directed evolution studies to essentially accelerate substrate specificity engineering. We have experiments going on right now to do this, and don't have any experimental data yet to share here. Another application that we're working on is essentially for using these models to help accelerate antibody maturation protein engineering. So essentially transfer learning the model, but just the amino acids that are at the antigen-antibody interface. So again, same self-supervised learning task, but instead we select residues that are within 5 to 10 angstroms of an antigen. So again, right here is a light and heavy chain antibody shown as light and dark blue, and then here is the SARS-CoV-2 antigen shown in yellow. We can select all residues on either the light or heavy chain of the antibody that are within the vicinity of the SARS-CoV-2 antigen to generate mass microenvironments. So we've collected datasets of all antigen-antibody complexes in the PDB and then have been transfer learning to this interface and then have currently are engineering antibodies for COVID-19. And I don't have any experimental data yet to show there. So I just want to thank you all for paying attention. I want to thank my computational collaborators, my graduate school mentor, Raghav Shroff, my software engineering mentor and co-founder James Loy, the experimentalists on the enzymes and proteins I showed, Hong Lu, Ibru Kayağrı, Simon Dolznitz, and also my professors and mentors here at UT, so Andrew Ellingson, Klaus Boepke, Eric Anzlin, and all of our different funders for all the proteins we engineered. And I'd be happy to take any questions. Well, I see one from 5013. Yeah, we were going to joke that we're a machine learning model designed to ask questions, but we're just a rheumatic company. I was just curious about the dimensions of the microenvironment that you used, and just as a follow-up to that, whether you tested different dimensions and saw their impact on your predictions? Yes, we use a 20-angstrom cube. Currently have models trained that use different dimensions. They, for the most part, do about the same. It's more— what changes more is, I guess, the mispredictions between them. So the wild-type accuracy, you know, about the same similar to how I showed for like the sequence similarities. They're all between 83-85, but it's more about, I guess, like the mispredictions, right? Like one of them mispredicted this versus that one. I don't really know what that means from the computer perspective. We usually figure out these things via engineering proteins and see, you know, which model type tends to give us better hits at the bench. Just kind of a semi-related follow-up, have you, have all of the proteins that you've engineered, have they all had experimentally solved structures, or have any of them been, you know, have you used homology models for any of them? So for this one, So for the serine dehydratase, they had been working on this for a while, and they have accumulated like, I think it was like 8 or 7, maybe 10 mutations by the time that we started working on them. So we just made a homology model, but, uh, and we were able to get a hit there. And then for— what was that? So was that an experimentally determined hydrotase structure that you then just made? Yeah, it's a homology model from that. Yeah. Yeah. Right. And then now we use AlphaFold. So here, we've been doing a lot of this. We've gotten a few positive experimental results doing this so far using AlphaFold and docking. So that's another computational structure. Yeah. Is your hit rate reduced with the homology models? Hard to say, right? Because different experimentalists will make different amounts of mutations. So like, it's hard to compare like one protein where we make 50 to 60 designs and another protein where we make 10 to 15, right? But we so far tend to get at least several mutations for each one. And we're able to stack, stack them together. It's all about having enough to go spend time stacking them. Yeah, it's hard to say like systematically, you know. I— we don't have no statistical significance, you know. Cool, thank you very much. Any other questions for me? I have one question. Do you think that the amino acid imbalance in your dataset— so I could see that you had more alanines than, say, cysteines or tryptophans— do you think that that in any way biases the model, and would correcting for that have any effect? Yeah, that's an interesting question. So if you go read our first paper, And if you go read the paper that we kind of built our work on, they essentially would sample all these proteins, see which amino acid was like least abundant, and then use that as like, okay, we're going to sample all the— all of that amino acid and then grab that many of all the other amino acids to have them all sampled evenly. And that actually gives you worse performance And you can kind of see that here. I think that was the improvement that we did. We went from this blue line to this red line was changing how we sampled it. Actually, it was both. I think it was going from here to here. The random sampling with reweighting, that gave us the improvement. And if you go look at some of the figures in this paper, we kind of show like by sampling via natural abundance rather than by evenly sampling each classes, the predictions are just, I guess, better. This kind of makes sense, right? There's a reason why all those amino acids are— that is their natural abundance, right? It's not a coincidence. Nature has taken hundreds to billions of years to come up with that abundance of each amino acid. So if you start like over-predicting like undersampled amino acids like cysteine, you know, more often than not, if the model predicts to put a cysteine here, it really isn't needed because cysteines aren't really observed that often. Well, but then what about the genetic code? There's a different kind of pressure that may also influence amino acid distribution, isn't it? Yeah, so that genetic code is kind of— it's kind of indirectly responsible for the natural abundance we see, but that genetic code is also optimized for that reason, right? There's a reason why I think only— there's only one codon for cysteine. Because you don't want to be constantly putting cysteines on accident. Okay, maybe, right? Because then you have a lot of proteins getting oxidized, forming disulfide bridges that shouldn't be there, and like unfolding or killing the protein, right? While other amino acids that are more permissive and you can be swappable between, they'll have 4 codons, and then that's why they have— they'll have a higher natural abundance. Yeah, but there's no true correlation between natural abundance and codon distribution. It's not a perfect correlation between natural abundance and what? Codon abundance. I mean, yeah, yeah, not all amino acids that have many codons are the most abundant. Yeah, I mean, I don't— I'm not a geneticist, right? I'm more of an enzymologist and a chemist, so I don't really know, like, how to respond to that. You're probably right. I just— we've just observed that whenever we corrected for that, the model trained and got a higher accuracy, was better to re-predict the wild type, and we've shown that these predictions also help us improve proteins in purity. Okay, it's a brilliant observation that I, I don't know what to make of it yet. Probably in the future. Yeah, like it would just take a lot of money and effort to be able to like train models with both and then systematically engineer a lot of proteins with it, see if that actually helps us forward predict. Okay. So one thing I'm currently doing is I'm trying to put together a lot of these DMS datasets to kind of start answering some of those questions, right? Just building all the data engineering pipelines to quickly benchmark against DMS datasets. Great work anyway. Thank you. Thank you very much. Any more questions? I just had one more question. Excellent work again. I'm just wondering if you had a chance to compare any of these results to what might have come out of maybe a simple model of coevolution or of evolution, just looking at a consensus sequence, for instance. Yeah, so I have not. I think, uh, like, that would be like a very, like, like a very bioinformatic-based project and like systematic comparison of these models to like other models, or, you know, to like maybe co-evolution, epistasis, maybe like even a simple linear regression model. Um, our lab doesn't really focus much on like bioinformatic type or like systematic analysis of these things. It's more about We have an engineering goal. We have a protein we want to engineer. We're more of an experimental lab, right? This, I think it's like me and like one of the students computational in this lab, and we just try to train models that like sample the data, the data in a way that we can make predictions for the protein engineering projects in our lab, if that makes sense, right? So that's not really like our lab's primary objective is not to like systematically analyze and correlate against all these things. It's more about can we actually make proteins and engineer them? Yep, gotcha. Yeah, there's only so much you can do with just a couple of people or limited. Yeah, it's pretty much just me that I'm in the computational office, and you can tell it's just me. Well, again, great work. Thank you very much. Thank you. What sort of, um, so I come from a computational lab, bioinformatics lab, like you just discussed. What sort of like metrics and visualization scores do you usually use to convince experimentals or convince yourself that a protein can actually be designed and engineered to be better? 'Cause I definitely work in one of those bioinformatics labs that maybe sometimes isn't quite as aware of those wet lab, like transitioning to the wet lab type designs. Yes, so usually whenever I'm training models and I'm deciding like which models to kind of move forward with. I, uh, so my background's like in synthetic organic, physical organic chemistry. So usually I'll like, whatever the protein we're trying to engineer, whatever like proteins we have engineered already that we solved structures for, I would just go see like, do we re-predict what we already know works? And I'll go look at like interactions that I know like make chemical sense And then I go and see, like, okay, does the model predict stuff that we know works? Like, there's like a salt bridge, right? Are you telling me to break this salt bridge? That would be a poor design, right? This is already a stable interaction. I don't want to break this. And most of the time I— so I kind of show this right here. And we tend to do this too. But a lot of the times whenever we give out designs, I usually customly tailor them where I will go through and like I delete designs where like I don't think they make chemical sense. So maybe it's like a crystal artifact, or you know, AlphaFold gave us a poor PDDT score in the local environment, so I don't trust the design, things of that nature. I kind of use my chemical intuition just from like years of doing organic chemistry to kind of filter the designs. So that's— it's very non-systematic, but we tend to just look at designs we've made in the past for proteins, and I just go look at interactions I know that shouldn't be broken, right? So like if I go look at, you know, this arginine to valine prediction, and I go see that this arginine is currently forming a hydrogen bond with an aspartic acid and it wants to break that, I'm gonna be like, okay, this model is not doing good. Hopefully. I definitely, definitely didn't quite catch why that was not good, but I'm going to say yes. Thank you. Yeah, so that makes sense. Cool. Thank you. Well, uh, there's two more questions in the chat. Chris, can I read your question? Yeah, thanks for a great talk, Danny. I was wondering, does MuteCompute take into account sort of bound waters and hydrogen bond networks that are mediated by them? So think about what you're saying about sort of salt bridges and things like that. Yeah, so I have, while I'm training MuteComputeX, One of the problems with water on MuCompute was how do we get like partial charges on water based on like what it's hydrogen bonding with, right? Different water molecules can be polarized differently. So, uh, in the MuCompute X stack, we're able to actually calculate all those values. And I have trained models where I actually— so Because we can make all the data on the fly and I just kind of store the data as metadata. When we're making all these boxes, I have a config file where I can just say water equals true, and then it would actually add all the water molecules that are surrounding all these residues. So you can think— but what I have seen doing that is by adding these water molecules these water molecules are crystallographic water molecules, so they're not necessarily what water is doing when the protein's like in the cell or soluble or like floating around, but it's just like how the water happens to crystallize in the structure. Maybe some of those water molecules like in the active site or like in internal cavities of the protein probably do represent what the water is doing in the folded active protein like within a cell, but usually I've seen whenever I take those models will train better and give me a higher accuracy like wild-type prediction accuracy, but they generalize worse and they correlate worse with like FireProtDB and other like experimental data. And they also prevent me from running predictions or inferences on AlphaFold or homology models that don't have any water. I hope that answers your question. Yeah, yeah, it does. I mean, I was thinking about water molecules that are involved in the active site and are, you know, present in multiple crystal structures of homologous proteins, so they're actually actively involved rather than just there as a crystallographic artifact. Yeah, so I would have to— I, I agree with you there. It'd be great if I can kind of tweak my data engineering to like automate deciphering those two, right? Like just don't blanketly add all waters or don't blanketly remove all waters, but look at the water's environment and then choose whether to keep it or not. But, uh, yeah, I just— I have not played with like this, like actively choosing in an automated way which waters to keep. But I do agree with you. I think adding those waters that are in the core, in the active site, or just within the actual protein scaffold and not just on the surface as crystallographic artifacts, I think would be an improvement to the data engineering stack to train these models. Oh, that's great. Thanks so much. Last question from Berta Rujales. Is there a version of these slides available for download somewhere? A lot of these slides— so a lot of the work I showed here is still— I'm currently starting to write up a lot of the new ComputeX stuff. So I just— my professor would prefer that I publish it before, or at least get it out on bioRxiv before just openly sharing it. But I think this whole presentation will be on YouTube. So I guess it'll be available there. One sense it'll be available. Cool. All right. Yeah, I expect to get probably a bioarchive out of a lot of these improvements. I kind of gave you a little taste of today, hopefully by the end of the summer. Great. It's always nice to see unpublished stuff and talk. So yeah, is there any more questions? I really enjoyed kind of sharing this and really great questions today from people from a lot of different backgrounds. That's the beauty of this series, is we have people from many backgrounds. Just going back to the question of the water molecules, how do you actually place the hydrogens in the water molecules