Scientist Stories: David Baker, Protein Design, AlphaFold, & De Novo Prediction of Protein Structure

Yes, you're fine. Yeah, so since it's a workshop, I kind of thought maybe rather than sort of the conventional seminar, I could sort of outline, sort of begin just with some high-level comments and then sort of figure out what to focus on. So like Jose said, we've been working on designing proteins from scratch for quite a long time now. And the basic— so the basic problem is— so what that means is not sort of elaborating on naturally occurring proteins, which is what protein engineering has largely consisted of. The distinction is, you know, everything I can tell you about today, we're building from first principles. Now, what those principles are First principles is kind of a loaded term, as you know, but there are basically two different approaches. The first approach is physically based, and it's based on, of course, the idea that proteins fold to their lowest free energy states. So if you want to design a sequence which will fold to a new structure, you have to find a sequence whose lowest energy state is the desired structure. And of course, sequence space is enormous, right, because you have, even with just 20 amino, natural amino acids per position, you have 20 to the 100th possible sequences for a 100-residue protein. So over the years, we've developed, and we and our colleagues have developed, a program called Rosetta, which does the sampling in sequence space, this huge sequence space, and also the sampling in structure space needed to design proteins using this approach. And much more recently, of course, deep learning is having a big impact on computational structural biology. And so here, of course, it's quite different. You're not starting with a physical principle. Instead, we have, for example, a structure prediction network that accurately maps from sequence to structure. And one can ask, if one wants to design a new structure, one needs to find a sequence which maps to that structure using the network. So whatever principles are in the network in that case are sort of distributed among the 10 or 100 million parameters of the network. So what I sort of put together a talk where sort of going through sort of examples of what one can do now with de novo protein design using the physically-based approach. And then in the second part, talking about how sort of deep learning is kind of transforming everything. And I just— some of the topics I could cover are just— and this is really mostly with the— the top ones are mostly with the physically-based approach. So I'm not sure if that's so relevant to the topic of the workshop. But for example, design of binding, conformational change, membrane proteins, self-assembly, nanomaterials, rotary machines, and then design using deep learning. But maybe, I don't know, maybe we could just pick a couple of these topics that people are most interested in, and then I could get to the deep learning stuff sooner, which is I think more focused on more the point of this workshop. So yeah, so what would people be most interested in hearing about?

I think that you should include design of nanomaterials and you should include deep learning for sure.

Yeah.

And then you can choose yourself a couple of the more interesting ones of the other topics.

Okay, that sounds good. That sounds good. Um, so I will do exactly that. Um, uh, so let's see. Um, why don't I, um, why don't I just skip down to talk about the nanomaterials and deep learning stuff. And then if we want to talk about other things, then we can talk about them. And does that sound good?

The floor is yours.

Okay.

All right. No, I understand. Okay. Well, maybe I'll just give one slide just to show you the sort of problems that we've been working on in these other eyes, in the other— for each area. So one of the problems we've been working on is the general problem of, given a protein target— and this is just an illustration with the coronavirus— design proteins which bind to it. So it's— and we try and frame this in a very general way. So you've got an arbitrary protein structure, and you point to an arbitrary site, and the challenge is design a protein which has high affinity and specificity for that, for that surface. And so they're kind of— the basic idea is we generate a large number of protein scaffolds, so we have lots of different shapes to play with, and these are all virtual at the beginning. And then we dock those against the scaffold, against the target, and identify those that are shape complementary. And then we design the interface for high affinity binding. And so then, so basically the design challenge, there's kind of two parts when you're designing the sequence. The sequence has to fold up to the structure that— and then that interface you've designed has to bind to the target tightly. And this works really well, and I'm just going to show you this one example, but this protein here is kind of remarkable. It's a 55-residue protein. It doesn't melt at high temperature, and it binds to the to the target with about 15 picomolar affinity, and it is— and we've done this now for quite a number of different proteins. So this general problem of designing proteins which will fold up into a structure and present an interface to bind an arbitrary protein target is one that, that's getting close to being solved. Now, the success— now, we fail a lot of times, so we have to make a bunch of different designs for any given target. But at least the folding and binding part of the problem for these simple proteins is pretty well understood now. And I'll allude to it later, but we can really improve the success rate using deep learning. And I'll just skip through the rest of this, but these things can have real utility. So this compound is going to be in clinical trials next year. So I'll just skip all this. And so just to show that there's some interesting applications out of the biological realm. So that was designing proteins to bind to protein targets, but we can also design proteins to bind to small molecules. So as you know, in the core of photosynthesis is the special pair of two chlorophyll molecules, which are held in— which are excitonically coupled, and they're in this huge photosystem proteins complex, so they're hard to study. So what Nate Ennis did was to kind of just take the two chlorophylls in their geometry in the photosystem and then design two proteins that— a protein homodimer that holds them in exactly that same position. And he was able to do that. So now it's kind of neat. So we have this sort of like laboratory system for studying the photophysics of two chlorophylls arrayed in this range, this arrangement. And then he can change the spacing of the chlorophylls and it's pretty neat. They are excitonically coupled. And that that definitely depends on the environment. So then in sort of thinking about nanomaterials, so we can take this protein that has— this dimeric protein that binds to two chlorophyll molecules, and then design interfaces with it, between it and another design protein that drive formation of this cubic structure, and here are the cubes here, and here's a an EM reconstruction. And so here each edge has two chlorophyll molecules on it. So we can start to look at, you know, start thinking about artificial light harvesting complexes and start looking again at the photophysics of bringing these different groups in close proximity in different arrangements. Okay, so I'm going to skip through this, and just wanted to— the only thing I'll say here is we can apply these same physically based principles designing not only soluble proteins, but transmembrane proteins, and when we design proteins that have pores and put them into membranes, they act as selective ion channels. And one of the interesting things, maybe I'll just say a few words about this one. So I told you that in, in the design calculations, what we're trying to do is find sequences whose lowest energy state is the target structure. And here's a case where there was a notable exception. So we were trying to make proteins that are all beta-barrel proteins that insert into membranes, and this is work from Anastasia Vorobyeva. And what she found when she tried to design these proteins is that we all, after the computational design step, we make synthetic genes that encode the proteins. And when she tried to put those proteins into E. coli, like we did for all the other examples I just sort of skipped through, she found that the bacteria just didn't want to make the proteins. And so what she figured out is that in an all beta— in a beta barrel, a transmembrane beta barrel, like the outer residues sticking out have to be hydrophobic. And in her sequence design calculations, she was optimizing the kind of the beta sheet propensity. So the problem is if in a cell you make a beta protein that's a beta— lots of beta strand-like structures that are very hydrophobic, what will happen is they'll aggregate, you'll get amyloid formation and so forth. So what she finally figured out is that what you have to do is actually just make these beta strands very low, have very low beta sheet propensity, so essentially the beta barrel only forms, they only become, have beta structure once they insert into the membrane. So that's an example where sort of had to think about the folding process as well, not just the final state. Okay, so thinking about nanomaterials, so one of the very interesting nanomaterials are obviously protein crystals. And, you know, crystals form all the time, but it's been very hard to program structures of crystals. And so, um, uh, recently solved this problem, and their approach was hierarchical, sort of taking advantage of sort of successive reduction of the degrees of freedom. So starting with the design monomer, first they designed it into a cyclic oligomer, doing this with two different monomers. So we have cyclic trimer and dimer, and then these are arrayed on the symmetry axes of a polyhedron, in this case a tetrahedron, and interfaces are designed between them, that direct formation of the, of the tetrahedron in this case. And then we dock these together. At this point, there's really just one degree of freedom once you put the tetrahedron on the symmetry axes of a cubic, a crystal lattice that has a tetrahedral sort of point group symmetry. And so they were able to design 3 quite distinct crystals in this way, which are very close to the design models. And this is kind of neat. So now we can sort of program 3-dimensional assemblies. And again, in this case, this is all energy-based. So each one of these interfaces at each stage, we're designing sequences to sort of that will— that are predicted to fold the monomer structures, then hold these complexes, hold each of these structures together. And so proteins, you know, I think one of the things that was you know, for Max Perutz and the others who solved the first protein structures, they're extremely irregular. But with protein design, we can actually— it turns out that irregularity is just something, is something that, that's nature's fault, not the polypeptide chain's fault. So Tim Huddy, a graduate student in the group, has figured out how to make proteins that are actually very, very regular. So, and this I think can make them much easier to study the physics of. So this is a he's designed proteins that are perfectly straight and perfectly repeating. So you can, if you make an assembly with them, for example, this protein square, you can make it bigger just by inserting and changing, expanding, increasing the number of repeats. These are just EM structures showing that the square is expanding. And he can make more exotic structures like, these train-track-like structures where you have, you know, a, a, a, one of these straight repeating proteins here, and then a C2 dimer with the C2 axis here. And you can see the pictures of these train tracks here. So this is much more modular than proteins are in nature, and so we're thinking we can really design a whole class of new class of protein nanomaterials using this approach. And this extends to 3 dimensions, so We can make cage-like structures, like this is an icosahedron, where the arms are built out of these straight extensible proteins, so then we can just expand them, and this shows— this just shows some experimental data on one of these cages showing that we can sort of systematically expand the structure. So, so far I've talked mainly about designing for protein ground states, but we can also design proteins that have multiple states, two distinct states. And so here there's like— this is again one of those repeat proteins. It's got this state, then it's another state where it's opened up, and we can drive, toggle the equilibrium between those by adding a ligand. And I'll just skip through This is the crystal structure of this helix, of this ligand, which is a helix in this case, bound to the open state. And we can use spectroscopy to follow the conformational change. This is DIR spectroscopy in the unbound state, where the labels are far apart, and then in the bound state, where the labels come closer together. And so we can, and in both cases, we have a quite well-defined structure. And we can now sort of get at concepts which have been around in protein science for a long time, like allostery, because we can— these— this ligand changes the shape of the protein, and if we make assemblies out of it, then we can actually drive changes in assembly by adding this ligand. And it's not by modifying the interface, it's sort of action at a distance, because the ligand is is changing the conformation of the protein. And I'll just give you one example of that. Here's a designed trimer which is in this unbound state, where the monomer is in the unbound state, and then if we add the ligand, that changes the monomer to be in the bound state, so it closes up, and that gives rise to a different shape, which now is now designed to be a C2 dimer instead of a C3 trimer, and we can see that kind of conformational change when we experimentally examine the proteins. And so the last example I'll give you of protein design using energy-based methods is sort of this, this simple kind of proto-nanomachine, and here we have an axle and a ring. Alexey Corbet figured out how to design these type of axles and rings and drive the assembly. And so what's interesting in this case is that you can— the axle and symmetry, the axle and rotor systems are both of cyclic symmetry. But what we find is the energy landscapes are very different when the symmetry matches compared to one it doesn't match. So here we have a cyclic axle, C3 axle, and C3 rotor. And as you'd expect, the rotational energy landscape has 3 pronounced minima. And in other cases like this one where we have a dihedral D3 axle and a C5 rotor, they are there's a symmetry mismatch. So you think about the rotational landscape, it's not— it's not the periodicity— is, is, is not— there's not like 3 simple global deep minima because the symmetries don't match. And so we get much rougher energy landscapes shown here, as shown here, without really deep minima. And when we look by EM, this is sort of— it fits. Like in this case, we have a very diffuse electron density for the for the ring, so it's as if the ring is undergoing rotational diffusion, whereas in this case we get much better defined structures. And in some cases like this one, we actually see discrete structures. Oops, sorry. So here we have a D4— let's see, this is a D8 axle and a C4 rotor, and we actually have two C4 rotors on. The axle is dihedral symmetry, so if there's a rotor here, there must be one here as well. And you can see here we're capturing two distinct states. There's an eclipsed state and a staggered state, and they're conformationally resolved. So, now we're trying to build in— we've actually built in catalytic sites, and the hard part here is the single molecule measurements, actually tracking to see if we're getting directed motion. We're also designing systems that will walk along a fiber, a designed fiber where the measurements are probably going to be more easy. That is a high-level overview of what become possible with energy-based design. Maybe at this point, I could take a few questions and then the rest of my talk will be about deep learning and it will be shorter than the part I just gave.

So how would you comment on how sophisticated these energy functions have to be in order to succeed on getting these structures?

Yeah, it's a good question. Well, the Rosetta energy function is not very sophisticated. You know, it's pretty much— I mean, they have to be relatively quick to compute, right? Because you have to do, you know, you're doing many, many thousands of energy calculations for each of these efforts, maybe millions. So basically the Rosetta energy function is pretty much like— it's basically a molecular mechanics energy function. There's a Lennard-Jones potential in electrostatics. There's electrostatics. It's implicit solvent. So it's— because just to make it tractable, and we want to calculate free energies, so we need an implicit solvent model. The only thing that's really different— well, okay. So the difference between a classical force, a molecular force field like Amber or Charm is, first of all, we— it's implicit solvent, and then we model hydrogen bonding explicitly, just because the hydrogen bonds are really important for biological specificity, and the simple electrostatic model doesn't really capture all the orientation dependence.

Yes, I have a question about your assembly, self-assembly for the rotary machine that you've shown there. So, I'm familiar with the bacteriophage motor, which is very well, probably the best nature-built nanomachine rotary machine there is. So there, I think, I mean, the question is about both about design and about the protocol of what do you do in terms— because this is a multi-component machine, right? So in terms of assemble to this accurate machine, do you have to pay attention to what do you express first, what are you putting first, and how much do you put in next? So the self-assembly process actually is called a self-assembly, but it was never a self-assembly. It's highly controlled in terms of timing of what protein is expressed first and the second. So they go in sequences in terms of assemble, at least for the bacterial flagella motor case.

Yeah, less and more complicated. Right. I think that makes— yeah, I think that's absolutely true. I mean, from, you know, these systems are just two different components, right? They have the axle and the rotor. We do have to pay attention to how we assemble the ring on the axle, and some of the ways we do it are outlined here. I think that staging self-assembly becomes more and more important with increasing number of components. We have systems where— I think the So the answer is, is the— one of the things that's most critical that we find is some of the components may not be that well behaved in isolation, so they have to be made in combination with other components that shield their interfaces. So I think that really thinking about the process of self-assembly will become more and more important as we go out to more complex materials made out of larger numbers of protein chains.

But in your hand, for example, if you just mix them together in the way that you do it, how— what is the yield? What is your yield?

Right, so that's a good question. The other thing that we're doing here is we're purifying after assembly, and so I think typically it would be around 50%, and then some fraction, like you could imagine that the that the ring could close without being closed around the axle in this case. So, you know, where it'd be by, by sort of selecting for, selecting out the complexes that form properly at the end, you know, it definitely, you know, that's, you can get by without 100% yield. I'll show you one, maybe I'll show one example of a slightly more complicated assembly that So, and this is one where we spent a long time trying to figure out the optimal way to assembly. So this is basically an octahedral nanoparticle made out of a C4 tetramer, and the C2 in this case is a naturally occurring protein. It's basically an antibody. When we mix these two together, they assemble to form an octahedral structure. Now, if you think about the symmetry of octahedral point group symmetry, here we have a C4 and a C2, but there's also a C3 axis, which is which is where, like in a cube, where it would be the diagonal going through the corners of the cube. And so, what we did was design a third component here, and that fits perfectly into this, onto this empty C3 axis as shown here. And this is now a cryo-EM structure of this assembly, which is a little bit more complicated because it has 3 components. Here's the C3, here's the C4, and here's the C2. So, now, in assembling these, forming the octahedron first, and then adding the, the C3, which we call a plug, turned out to be very important. So this is a case where order of addition mattered. We can also make this environmentally dependent, because we can design particular components like this primer to come apart, come off at at low pH. So I think you're right. I mean, we're definitely, definitely, as we get to more sophisticated nanomaterials, thinking about assembly is going to be very important.

So I had essentially the same question, which is what is the yield? And I'm actually very surprised that you say it's as high as 50%. Do you specifically design things to not have, like, weak non-specific interactions? I mean, I thought that you know, part of the success of DNA origami was in part mitigating that. And I mean, are you doing like complicated annealing cycles, or is this just you mix things together and you get 50%?

Yeah, well, it is you just mix things together. I mean, it's because we're designing— I would say the biggest failure mode we have is if you have too much exposed hydrophobic surface in the components. They will self-associate rather than associating with their partners. So, what you— and basically one of the— so, it's sort of implicit negative design. So, you basically try and avoid any excess hydrophobic surface, which could mediate off-target interactions. And, well, it's funny, because just yesterday I was talking to my student who was working on this, who was complaining bitterly that he was only getting 50% yields. So, you know, it all depends on what you're— where you are in this. But, you know, DNA origami is a lot more common. I mean, you know, you have— those are much more complex. The number of components is really huge in those. You think of all those primer strands. So here we just have 3 proteins with pretty robust interfaces between them. And so I think it's a simpler process.

Hey, David. Neil Woodbury.

Hi.

Question that I'm sure you've thought about, but is something that, that I think about a fair amount has to do with the sort of topology of these spaces. And that is, if you think of something, let's take your first example, which is the picomolar binder to the spike protein, for example. How many structures in the 20 to the 50th possibilities out there could you have come up with? Do you have any idea about that?

Yeah, that's a really good question.

Is it a million? Is it 10 to the 20th?

I mean, well, yeah, that's a really good question, and it's a little bit hard to answer because we, you know, we're basically doing an op— we're first, we're kind of doing this extensive optimization process on the computer, right? Because we're generating very large numbers of scaffolds, and then and then going through this sort of selection for structures which have shape complementarity and then good interfaces. So I think it would be hard to— I mean, what we can say is at the— we can basically talk at the— so in the tail of the distribution, the part that really is favored to bind, there definitely are a number of solutions. We get a number of different sort of shape proteins that fit. But of course, our process is far from perfect, so there could be far more. If you said that our physical model was perfect, then you might be able to start trying to get an estimate of the answer there. I would say, and also it's going to depend on what your standard for difference is, like how you discretize the space. But our physical model is really pretty far from perfect, so I think that would be pretty crude estimate, but I would guess, you know, on the order of, I mean, hundreds of very distinct structures. But I mean, also, and I'll get to this when I talk about deep learning, is usually it's just a part of the protein that's binding to the target, and then the rest just has to support that in that particular geometry. And so that, there, the possibilities are really enormous.

Okay, that's a point, good for transition. We have lots of other questions here, but people are very curious about your deep learning part of the story. So let's move Okay.

All right. Okay. So I think I won't go into the details of network architecture here. So this is the architecture. So we've been— so basically what happened, I think you know the story, that DeepMind announced these amazing results at CASP a couple of years ago, but they didn't really say much about— it wasn't clear whether it was ever going to become available or And they didn't really say much about how they did it. So we got kind of intrigued, and Min Kyung Bak sort of developed RosettaFold, basically building off their ideas. But the implementation turned out to be very different because we didn't really know any of the implementation details at that time. So since then, we've improved it. She and Frank DeMaio have improved it quite a lot. And one of the things that's interesting about RosettaFold that's distinct from AlphaFold is you basically input sequence information, sort of any kind of 2D structural information, and then template coordinates, and then those kind of propagate along all the way. So in AlphaFold, you have the sequence track and sort of a contact map track. In RosettaFold, you have a 3D structure track all the way through, and that turns out to be very useful for design. So in terms of performance, there's no doubt AlphaFold 2 was quite a bit better than RosettaFold. The new RosettaFold is not better than AlphaFold. It's comparable, but it's better than— so DeepMind released this AlphaFold Multimer program, which isn't quite as good on monomers. And then on complexes, the new RosettaFold is about as good as AlphaFold Multimer. So it seems to have the better features of the two. But of course, the real question is, are there use cases where people find it useful? And so, okay, so how do you design proteins with something like this? Well, I'll describe several approaches. And the first approach is kind of getting at Neil's question. So what if you just do completely random searches in sequence space, and you search for sequences which are predicted to fold by a network into a very well-defined structure? And so here's— so this, you take a completely random sequence and you predict the structure. This is a contact map. So these are contacts between residues that are close together along the sequence. Contacts between residues far apart, they're very few at the beginning. This is a random sequence. Random sequences don't typically fold. Now we do this Monte Carlo optimization in sequence space, or we can do gradient-based optimization also. And then after some number of of Monte Carlo steps, you end up with sequences which are predicted to fold strongly to structures. But the structures are really all different from each other because we didn't specify anything about the structure. We just specified that we were just rewarding sequences that fold to some structure. So we've solved the structures by NMR and crystallography of several of these, and they're quite close to the design model. So apologize for another horrible network architecture. So when we started doing this more and more, we found that the sequences, that the structures that came out were very interesting. But sometimes the sequences had some obvious pathologies. And that's because we're kind of doing, we're kind of optimizing over the inputs. It's kind of like activation maximization. We're trying to find sequences that give really strong inputs. So we're kind of— there's a lot of overfitting that happens. So what we did was to develop a network specifically for the problem of, given a protein backbone, predict a sequence which will fold to that backbone. And it basically starts with a very detailed description of the structure, and then And then the sequence is sort of autoregressively decoded given that structure. And this is— so we've worked on Rosetta for all these years, and this is a little daunting. So, okay, this is the standard test where you take many, many native structures and you predict sequences for them. With Rosetta, we would be doing Monte Carlo in sequence space, trying to find the lowest energy sequence for the structure. And then we have the method I just described. This is just looking at sequence recovery at different burial levels. These are more exposed residue positions and these are more buried positions. More buried positions, this is the fraction of the native sequence that you recover. More buried positions, it's like a jigsaw puzzle. Rosetta is almost as good as this neural network, but at all other places, the network is just much better at recovering sequences. And this holds across a very wide range of different types of proteins. So what we found is we could take proteins, take sequences that we had generated using this hallucination process, where we got to more complex problems, the amount of proteins the bacteria produced were very, very small. When we redesigned the sequences, the success rates went way, way up. We solved structures and they came out right. And so basically, we can now go on to more complicated things. And so now I showed you the hallucination where you just make a random sequence and you optimize folding. And just to show you how far this has gone, this is now— was a rotation project back in winter quarter. So here we take— We pick a length of a protein and a number and a cyclic symmetry. For example, in this case, threefold C3 symmetry. So we generate a random sequence, copy it 3 times, predict the structure, ask whether— and then in this hallucination step, the loss function, not only how well does it fold, but does it fold to a C3 symmetric structure? And we optimize sequences for this. And then we— once we get the structures out, we resign them with this MPNN network I just described. So it turns out these proteins crystallize kind of miraculously, so we've been able to solve crystal structures of 7 of these. And again, we put nothing in about the structure, not the secondary structure type, not nothing, just the length and the symmetry. So these are like C2 structures, these are C3, these are C4. So you can see that they're kind of complicated structures. You know, look at this one, for example. And the crystal structures match the hallucinations, the computer hallucinations, really, really closely. So then, encouraged by this, we went even larger. So here, this is taking a sequence and then replicating it, in this case, 33 times. And then doing this op— this hallucination process. And what came out were sequences that when we made them in the lab and looked at their structures by EM, they turned out to fold to almost exactly the structures that we had— that had been hallucinated. And there's a cryo-EM structure of this kind of huge ring with 33 identical repeats. And you can see there's different structures here. Okay. So basically, this is like just blind exploration of protein structure space. But we can use this also to design enzymes and design functions. So this is a luciferase reaction. And so, what we did was to generate scaffolds using this hallucination process with pockets that are the right size and shape for binding the transition state for this chemical reaction. And then, in this case, we were still using Rosetta to design the side chains for high affinity binding. And it's cool, we got things out that are very active luciferases with Kcat over KMs in the 10 to the 6 per mole per second. Level, and because we've designed this really close complementary pocket, the specificity for different substrates is much higher for this design than it is for the, for naturally occurring luciferases. Okay, so just the last few slides on sort of more general function design, and this, Neil, will get almost exactly to your question. So So let's say you have some functional site, like a binding site that's made out of these structural elements, and you want to design a scaffold which holds those in place. And so the purpose of the scaffold is to fold up to a structure which presents those two elements in exactly the right relative orientations. And the— so the first way we developed to do this is this hallucination approach where we add not only— we assess not only how well does the thing fold to some structure, but what does this— does the structure actually contain this structural motif? And so here's an example with taking a big protein like this is respiratory syncytial virus, taking motifs like this, and then fixing those and asking the hallucination process to build a a scaffold around it. So, so now, so now, Neil, this is almost exactly your question. So we have this binding function, and then what we find is there's a lot of different ways of scaffolding it. Here I think we could actually count. Um, and here's another binding motif where we come up with different— where the hallucination process comes up with different types of scaffolding things. And we make these things in the lab, and they actually fold up and bind the antibody, which is pretty cool. Um, and we can do the same thing with enzyme active sites, or fix the site and then hallucinate scaffolds around them. So the second way in which we can solve this problem is sort of the analog of sort of autocompleting of sentences. So if you type a few words into a word processor, it will autocomplete and fill it out. So we can train RosettaFold to do this. And so basically you can train RosettaFold RosettaFold was originally trained, you give it the sequence and predict the structure, but we realized we could turn it backwards, kind of like we did with Rosetta, and give it structures and have it predict sequences. And more specifically for this task, we could basically give it examples where we delete segments of sequence and structure both, and then RosettaFold had to learn to fill them in. So we call it— this is inpainting by analogy with sort of similar things with images. And so here, basically, we just give a functional site and we ask the network sort of to autocomplete the structure. And again, we get many different types of solutions. And when we actually look at the structures, when we make the proteins in the lab, they actually have the design function, which is binding metal in this case. So, okay, well, so that's basically where we are. And so there's a kind of really interesting question now about how will the physically based methods come together with the deep learning methods. And I would say there's an uneasy truce between them right now. So we'll see how that evolves. So let me tell you about the people who did the work. Longxin and Brian developed sort of this protein binding work. I talked about Alexei. And Jesse did the work on the molecular machines. Ivan and Sam and Tamuka, with help from Guy's lab, did the work on the unconstrained hallucination. Alex, Ilias, Sam, Basil, and Lucas made those symmetric oligomers. Eustace Doporos developed the MPNN. The luciferase was designed by Andy and Chris. And Ju and Sydney and Doug and Sergey developed this sort of constrained hallucination approach to functional site scaffolding. And Min Kyung and Frank are— sorry, David and Joe developed the inpainting sort of autocomplete approach, and Min Kyung and Frank are developing RosettaFold. So okay, I will stop there, and let's go back to questions, which is more fun.

Okay, just one, a couple just technical questions here. When you do the elucidation process, you still use the Rosetta force field for it?

That's how you No, no, the hallucination process isn't using Rosetta at all. It is— what we're doing is we take a network like RosettaFold or AlphaFold, and we— so we start with a random sequence, we predict the structure, and we have a measure—

when you predict the structure, you predict with what? We're using—

well, we've done it with different networks. We first did it with a structure prediction network we had developed a few years ago called TRRosetta, and then the work I showed you today was done with both RosettaFold and AlphaFold. So the network is kind of interchangeable. It's just an oracle, which tells you how well the sequence is going to fold. So for example, the cyclic oligomers were made using AlphaFold. So we generate a random sequence, copy it 3 times, feed it into AlphaFold. We get a predicted structure out. We assess how does it have cyclic symmetry. And how confident is the predicted structure. And based on that, we do— we're doing Monte Carlo optimization. The functional hallucination work was done with RosettaFold, but we can— we kind of use them interchangeably.

Okay, that's, that's one that maybe you can help me on. The functional part, we use the, the ensemble of structure that you predict for hallucination, or you redo the entire thing around the ligand, right?

Right. So, so no. So in the, in the functional part It's basically that the scoring function has changed. The scoring function in the free unconstrained hallucination, the scoring function is the only score is essentially the prediction confidence. How confident is the network that this in the structure that is predicted? We're optimizing the sequence to have maximum so that prediction confidence is maximized. In the functional site hallucination, we have an additional term, which is to what extent is— so we specify a functional motif, and what's the RMSD, essentially, of the closest match to the functional motif in the hallucinated structure? So basically what's happening then during the Monte Carlo optimization is it's trying to find the structure which is confidently predicted and contains the functional motif.

But you're predicting new structures, or you get that from the pool of structures that you predict before?

These are all new. Brand new. They're made in— it's basically, it's each hallucination is starting from a completely random sequence.

Okay, we have time for questions. Thank you, Dave, but let's continue.

Hi Dave, this is Amarda. I really love the connection from sequence to structure. If I understand correctly, the model that you have put together, it's not end-to-end, right? So that's the first part of my question. So once you get to a structure, then you have these sort of separate Monte Carlo-based optimization, but basically you are trying to get to a better structure for these particular random sequences. That's question number 1. Question number 2, is there interest— and I can ask this because I don't intend to venture in that domain— Is there interest in sort of pairing the sequence generation with a structure-based critic? So if I make myself clear, rather than start with a random sequence, why not have a sort of a generative network to design sequences and then to constrain that network in its generation process with a critic via reinforcement learning or a discriminator or something else? You could put RosettaFold in there. In order to really sample joint sequence-structure pairs?

Yeah, okay, well, those are both good questions, of course. The first question is, I described two approaches to do the functional design proteins that contain a functional site. The first one was— is not end-to-end. This constrained hallucination is really activation maximization. So you're repeatedly going through the network and using it just to evaluate a score, which you then optimize by an outer Monte Carlo loop. The inpainting approach is end-to-end. You specify your structural motif, and in one forward pass to the network, you get out the sequence and structure. So it's because the network's been trained for missing information recovery. So it's basically like sentence completion with a language model. Or image, you know, if you delete a part of an image, you fill it in with an inpainting model. So that is end-to-end and very fast. The second question is, yes, well, you're very astute. So we actually are developing GANs now that use RosettaFold, that start with RosettaFold as both the generator and the discriminator. And we're fine-tuning from there for exactly the reasons you allude to.

Hi, David. This is Tess from MIT. Thanks for showing all these super fun and inspiring examples. My question is mostly for the first part of the talk, but also applies to the second. And it's about the initialization of designs. So when you decide that you want to make, for example, this rotor, and someone sits down to the computer to design it, what do they need to pre-specify before you start enumerating scaffolds and doing the optimization? Optimization? And then kind of following that up with the deep learning side, where are those models at with respect to designing these sort of larger assemblies?

Yeah, it's a really good question. And I think your question gets at really a key point that is not often appreciated, which is that you can have design programs that design sequences for structures. But a lot of it, the questions are like, if you're going to design a rotary machine, what should the parts— what should the architecture of the parts be? And so there is no program right now which— and that's like a human expert or creator. So the programs are just tools, but to design something new like that, like a rotary system, it's like you need an architect who says, well, the ring should have roughly this architecture and the axle should have this architecture. And so that's really a human expert that's making those decisions. So in the second part, as far as designing assemblies, I showed you that we can use this hallucination process to design giant rings. And so in fact, what we're doing now, I just was speaking to Alex who's working on this, is we're using the deep learning methods to make the parts, but we're choosing the shape of the parts still. He has a higher-level concept in his mind about how they should fit together. So I think these tools are— I mean, I think the closest example to sort of having something that just solves the problem is like this functional site that we— so in the case of the functional site stuff I talked at the end, you specify the functional state, and the hallucination or inpainting just does the rest. And that could not have been done before with the energy-based approaches. You had to specify the architecture. So the deep learning methods definitely make it so you have to specify less than you did before. But still, in terms of the overall picture of what you're trying to do, that's still the human designer. Great, thank you.

Let me ask you to expand a little bit on two topics here. One is when you come from the energy models to deep learning. Tell me, are you just speeding the process? Are you able to design new things you're not able to design before? Are you doing better? And are you learning anything new?

Yeah, well, those are very good questions. I would say it's gotten more accurate and faster with the deep learning methods. And are we learning anything new? Well, that's the— of course, the thing about deep learning is that there is, you know, all the information is spread out through the millions of parameters of the network. So I would say I think you kind of— it depends on what exactly you mean by learning, but you come up with solutions to problems that you hadn't seen before. But in terms of the physics of folding, things become more obscure because we don't know exactly what the network is containing.

I see that's interesting. That's been a topic of conversation on several other issues. Maybe someone wants to follow up on it.

Just very specific. I think one chart when you show the comparison between the original energy-based approach and the RosettaFold, you see a significant difference And when it's— I think one end, I forgot which is the more buried, or— you see a significant gap opening up. But do you have any understanding of what's going on that causes that gap in terms of holes?

Yeah, I think we do. We have— I think we do. And so there wasn't a gap in the most buried positions. And in the buried positions is a place where the physics is the clearer, right? Clearest. It's really dominated by sterics. You want close packing, and it's dominated by packing interactions, which are really easy to model. In the less buried things, the physics is— it becomes much more complicated. You have interactions with solvent. You have— there's issues about aggregation and avoid— and so there's all these complicated things going on, and I think it's hard for a physically based model to balance all those effects harder than a deep learning model, which is just explicitly trying to reproduce what's seen in native structures. So like, if you think about completely exposed position, there the interactions are with solvent, and those are obviously going to be harder to model than the interactions are with, you know, within the protein where you know where everything is.

Hi there. This is Rafa Gomez-Bombarelli from MIT. And you mentioned the inpainting was end-to-end because it was trained on sort of 3D geometries to sample sequences. And then there is sort of other models that go from sequences to the fold. Are these bidirectional? Is this something— it looks like there is sort of a one-to-one mapping back and forth. Is this something that you folks are exploring, sort of how to have a single model that would go— flow-based models, for instance, go both ways, right? So are there invertible or bidirectional ways to think about this?

Yeah. So the way that RosettaFold is configured now, you can specify any amount, any starting information you have. So you can specify the sequence. You can specify the structure. So think of it as a joint language model, joint sequence structure model. So you give it whatever information you have, and it will complete the rest. So it's not really a flow-based model, but it's— It's maybe just a higher level. It's like a protein language model in a way that incorporates both sequence and structure. I don't know if that answers your question. We should certainly follow up on that offline.

It's conditioned on both at the same time. So you can sort of give it, mask all of one or all of the other or parts of one or parts of the other and sort of it will autocomplete everything else.

That's exactly right. And that— but I don't think that's quite answering your question, but that's, that is, that's basically how it's set up. Thank you. All right, thanks. Well, happy to follow up offline as well. And yeah, have a very successful workshop.