Scientist Stories: Simon Kohl, Highly Accurate Protein Structure Prediction with AlphaFold

All right. Hi everyone. Welcome to today's event. Great to be back after more than 2 years, I think, with an actual in-person Heidelberg AI event. I don't know, maybe many of you are new here in the meantime, joined DKFZ or Heidelberg, and this is your first event. So usually we also have socializing events afterwards with snacks and drinks provided to get in touch and Foster the community. This is not possible yet, but at least we can have the talks. And yeah, please look out for future events. And if you're interested in beers and pizza and meeting other AI enthusiasts, then yes, you're right. You're at the right spot. All right, so this event today is a feature of Heidelberg AI, also the Alice Unit Heidelberg, and the Data Science Seminar in Heidelberg. These are all kind of seminar series If you're interested in data science topics, you can check them out. Also, the data science seminar, if any of you are PhD students here at DKFC and you want to collect credits for this event, Doreen, where are you?

Here.

She will distribute lists and you can put your name if you want credits for your attendance. Also, Heidelberg AI is organized by the Medical Image Computing department led by Klaus Meyerhain, as well as the Interactive Machine Learning Group led by myself, and we together organize this meetup. And we are hiring, so we're looking for a PhD student at the moment. So if you are looking for a position or know anyone, please get in contact with myself. All right, so today's event is, yeah, we have Simon Cole as a speaker. And I met Simon when we studied physics together in Karlsruhe. We actually did the master thesis in the same lab, and that's also where we discovered our interest in machine learning. And then we looked for a PhD position and ended up here in the same lab at Klaus Meyerheinz's group at DKFZ, started our PhD together. And it only took 3 or 4 months into the PhD that our ways kind of parted. And it was one faithful night actually in this exact lecture hall where a professor from DeepMind was invited to hold a talk. And he was very, or is a very well-known figure in our field. And he was holding his lecture about his research. And it was Simon from the audience after the lecture asking, okay, so that's all interesting, but what are the future steps? Like what do you think is the most interesting direction for your research field, and the professor says, well, actually, that's funny because the other day I read a paper from Dekaev-Zed, actually from Klaus's lab, that I think is a really great idea, and I really think we should try that. And turns out this paper that the professor was talking about was Simon's paper, which he published only 3 or 4 months into his PhD, and then one thing led to another, and now we are here. Simon is senior research scientist at DeepMind. And so around 3 years ago, he told me that he's now not working on medical images anymore, but on proteins. And but it went, it was quite silent all the time. Like, yes, he's working on that. And I was assuming lots of people at DeepMind are working on that probably, but you didn't hear much about it. And that should make you suspicious, I guess, because maybe something big is going to happen. And it did happen last year. And I guess that's why we're all here in this room today. And now we have Simon to tell us what happened. Simon, what the hell happened?

Yeah, thanks so much, Paul, for this extremely kind introduction. I'm really excited to be here in Heidelberg and at the German Cancer Research Center where, you know, as Paul said, we did our PhD together. It's really not that long ago. It feels like kind of yesterday, and so it's really nice to return and present to you today about this work on highly accurate protein structure prediction with AlphaFold. So I thought it makes sense to give a brief introduction first. I'll be talking about why, you know, DeepMind is working on that, and then also a general introduction to the problem itself. So first of all, a central part of DeepMind's mission is to solve fundamental scientific problems. And one such very fundamental challenge is the prediction of the 3D structure of a protein sequence. And therefore, this is, you know, a very good fit for the company mission and is why we essentially worked on this. Also, AlphaFold, that obviously I'm going to be talking about, is our, you know, proposal of a solution to this problem. Now, let me talk about what proteins are just to kick things off and be on the same page. Proteins essentially are these molecular nanomachines, if you will. Like, you will have heard about nanomachines, but really nature has been engineering for these nanomachines over billions of years already, and they are really essential to life as such. They have many functions, from structural functions, for instance, to our hair, to our immune system. You will have heard about the SARS-CoV-2 spike protein, for instance, I trust.. And these chains of amino acids that are proteins fold up in 3D space, take on a 3D shape, and this 3D shape really, to a first approximation, is what lends them their function. And that's why protein structure prediction is such a fundamental problem in biology. Let's go into some more depth to sort of, you know, again be on equal footing. Proteins are these long molecules, we call them chains of amino acids. And nature really sort of chose to make proteins out of 20 fundamental building blocks. So there are 20 different amino acids, and from a computer scientist's point of view, they are therefore strings over an alphabet of 20 letters where each letter is an amino acid, or interchangeably, a residue, as we call them. Now, each letter designates a specific collection of atoms and their associated bonds. As I said, they're chains, and therefore these amino acids are connected up along that chain. And where they're connected up, we speak of backbones. That's their backbone structure. But then sort of dangling off to the side of the backbone, what we call side chains, and they're like, you know, small collections of atoms essentially specific to each amino acid, and they are largely responsible for the chemistry, if you will, that those proteins carry out. And then of course, I trust you would have heard of that, our DNA sequences uniquely and directly encode the protein sequence. So there's this one-to-one mapping from DNA to proteins, essentially. Okay, so with the introduction to proteins out of the way, let me motivate why we want to predict the protein structure. So in essence, experimental structure determination of protein structures takes months to years. It's really this very tedious process. Sometimes it's not even possible because you can't get the proteins to crystallize, for instance. And so for that reason, if you were to be able to predict the structure reliably in silico from just the amino acid sequence alone, that'd be a huge speed advantage and would enable many new things in biology as a whole. Sort of all of that is sort of encapsulated in that figure of the bottom of this slide. In the upper branch, what you can see is all the steps involved of going from the very left, which is the protein-coding DNA, to the very right, which is the protein structure you may be interested in. And all these intermediate steps, they're, you know, like I said, tedious. There are pitfalls around them. And again, it can take very long to arrive at an actual experimental structure. And therefore, what we set out to do is to go to bypass all of these steps and go directly from essentially a DNA sequence to the structure itself by, um, yeah, engineering, building, and training an AI algorithm that predicts the structure from the sequence alone. Okay, so AlphaFold was evaluated and sort of, uh, yeah, introduced, if you will, at CASP. And really, when working on fundamental problems such as these, what's really crucial is to have a clear success metric. Such that you can iterate in silico on that problem. And fortunately, the protein structure prediction community had provided us essentially with the means to do that. They had established a biennial structural assessment of structures that's called CASP. And this CASP assessment involves predicting recently solved structures that are therefore not known to the community. They're really held out. And therefore, this is a unique and, you know, fair testbed to test structure prediction algorithms against one another. And at CASP14, AlphaFold was the top-ranked method, achieving consistently high accuracy. And that was in 2020 by now. So at CASP, which is held every 25 years, AlphaFold achieved a median accuracy of 92.4 GDT, which really is sort of a metric that you can take to be an accuracy metric. It compares predicted versus ground truth structures, it ranges from 0 to 100%, or 100%, you know, would be perfect. And therefore, reaching anything above 90, you know, really was considered a breakthrough. If you look at this slide, sort of before AlphaFold 2, the computational performance was hovering in the, in the 40s, if you will, which really, you know, goes to say that it wasn't very reliable. And then AlphaFold, yeah, achieved an accuracy above 90%. And in response to that, AlphaFold was recognized as a solution to the structure prediction problem by the CASP organizers themselves. All right, I wanted to, you know, put some meat to this and show you two protein examples that were given as a prediction task at CASP. So this particular one is a coronavirus protein structure that again was held out and not known to the community. In green, you can see the ground truth structure that was determined experimentally, and blue was the structure that we submitted at CASP predicted by AlphaFold. And really, the reason I'm putting this here is to show that the two structures in this case really are basically almost indistinguishable. Like, by eye, they look almost the same. It's hard to tell which is the prediction, which is the ground truth. And that goes to show, you know, how good AlphaFold did in this case. And this particular one corresponds to a GDT of 87, which again is this structure accuracy metric. Now, this is another example that was posed as a task at CASP. It's an RNA polymerase. It's a really large protein consisting of more than 1,000 amino acids. Before AlphaFold 2, computationally, it was really hard to even try such large structures. Let alone sort of being successful at it. And again, sort of from this overlay, you can see that the prediction and the ground truth is very close and a very good prediction in this case. Also on this slide, I'm highlighting 3 crops of subunits, structural subunits that are called domains. And even on the domain level, these sort of 3 smaller GIFs on the lower right-hand side, you can see that the structural match is really good. All right, so with the introduction out of the way, I wanted to give a brief overview of the, you know, bits and pieces that I'm going to touch on. I'm going to be talking about how AlphaFold works, how AlphaFold understands proteins, how to interpret AlphaFold's predictions, also how you and really everyone can access AlphaFold, and then give some highlights of the things that the biology community was already able to do with AlphaFold. And lastly, I'll wrap up with a brief piece on AlphaFold protein interactions. All right, let's talk about how AlphaFold works. And really, I want to kick off this section by a small general piece about inductive biases in deep learning. So what we mean by inductive biases is concepts that we already have about the structure of a problem that we would like to sort of encapsulate in the architecture that we're building. And the reason for that, for that desire, is to make it easier for a model to reason about a problem. So it doesn't really have to sort of from scratch get all of the structure of the problem, but we can attempt to build it into the model itself. And I'm— I have 4 examples here that bring this point home. So the first one are convolutional neural networks that are very popular in computer vision. That the sort of data, the, you know, image data, is structured in a regular grid. And that's the thing we're making use of in building the architecture, where in convolutions information really flows from local neighborhoods to local neighborhoods. Then another inductive bias is the one that's used in graph neural networks, where we already know the fixed graph structure of the problem. That's, for instance, the case in recommender systems or on machine learning for small molecules, say. And we really try to, in these kind of cases, bake in the fixed graph structure of the problem. And information in that type of architecture can flow along these fixed edges. Now, in recurrent neural networks that are popular in language applications, data typically is an ordered sequence, and that is reflected in the architecture where information can really flow from the left to the right only. And I would say fairly recently, a very now, now very popular module has evolved, which is the attention mechanism, where again it started in language applications but because of its generality is increasingly used in other areas as well. And in this type of model, the data is assumed to be an unordered set essentially, such that every point in a in the set can essentially communicate with every other point. And this information flow is dynamically controlled by the neural network, by learned keys and queries. That is one of the inductive biases that we've also put into AlphaFold 2. There are several others, and in the next few slides I will detail exactly how we went about doing that. Um, yeah, are we taking questions now? Are we going about Yep. Okay, great. That's, that's it. After, I'll be happy to take questions. Thank you. Um, okay, so how we went about putting protein knowledge into the model is exactly by means of such inductive biases that are not really hard-coded in, but still ingrained in the model in a sort of soft way, if you will. And what we wanted to do in this case is put our knowledge about physical and geometric insights into that model and not just as a process around it, really into the architecture. And so one of the things that we've done is de-emphasize the sequential order of the amino acids, really to reflect that any residue or amino acid can talk to any amino acid in that protein. That's because really we want to infer sort of the graph structure, if you want to put it that way, of the structure of the protein. Sort of reasoning about which residues are close in space and then iteratively building that graph while you reason about it. And I'll give some more detail on the next slides how exactly we've achieved that, but that was the higher-level goal. Another very important point to make is that our model really heavily relies on evolutionarily related sequences, and this is such an important point that I've sort of dedicated two slides to this to give you an intuition why that is helpful and why that really helps with predicting the structure of a protein. So starting with this amino acid sequence that we want to predict a structure for, for instance, physics is, is really what carries, what carries out the folding process, if you will, and brings about the final all-atom structure of that sequence. Now, Throughout evolutionary history of a species or organisms, evolution selects for function while it mutates the sequence. So when function is conserved and sequences are mutated, what that typically means is that to carry out the same function, the structure will have to be very similar, if not the same. So given a, say, stack of evolutionarily related sequences, we can then attempt to computationally infer the structure. And that's really fundamental to AlphaFold 2, where we exploit that fact that, you know, this stack of sequences can tell us about distances in the structure space. Like I said, I'll have a second slide on that to bring that point home. So again, starting with a sequence that we might want to predict the structure of, we can perform a genetics database lookup so that, for instance, we'll find similar sequences across different species, as pictorially indicated here. This sort of stack of sequences we call a multiple sequence alignment, or MSA for short. And now to the intuition. Essentially, when, um, when, when you have co-evolving columns, so co-evolving positions in the sequences, that typically means that these residues are in contact, and that is really very valuable information in structure prediction. And the intuition there is, if you think of a charged bond, if you were to mutate a single charge away to, say, the opposite charge, that bond would break. So therefore, this bond can only be retained across mutational changes if both change together. And that's the cues we're getting from these correlations that we call coevolution in this MSA stack. Conversely, if amino acid positions, columns in that stack, are conserved, that typically means that they are either positions on the outside of the protein or on the inside, inside of the protein. And that's because hydrophobic amino acids tend to be on the core of the structures, and hydrophilic ones, the ones that are, you know, water-loving, they tend to be on the outside because proteins are typically engulfed by water. So all of this is very valuable information when you're trying to predict the structure. Okay, so I'll now sort of build up how AlphaFold works architecturally and start with talking about the inputs that AlphaFold takes. Now, I already spoke about, you know, obviously the target sequence itself and then the MSA stack that we get by doing a genetic database search. A third input to the model is sort of an outer product of the input sequence. And that really kicks off a pairwise representation where the model will then be able to explicitly reason about pairwise relationships between each amino acid in that sequence. And then as a fourth possible input, we have what are called structural templates. Those are structures that are already known that are thought to be similar to the one that we're probably inferring. Now, this is not a novel idea. People have done that before to model protein structure prediction. The thing that's different here is that we've enabled the model to attend to these or discard them. So it's sort of in a learned way, it can make use of these structural templates, but it's not relying on them. So often we find that we don't even need them at all. The MSA is enough input for the network. But yeah, if you have them, you can sort of supplement your inputs with those structural templates. Right, so those are the inputs. Then those go into the main part of the neural network where really we're operating on two different stacks of representations. So the MSA inputs, they are fed into and build up what's called— what we call an MSA representation. And then in a second stack, we have the pair representation where again, this is meant to be a learned way of reasoning about amino acid pairs in that sequence. And then both of these stacks are in a crisscross fashion updated, which I'll detail in a little bit. This part is done by the network trunk, as we call it, the main part of the network that we deem— that we name— gave the name Evoformer. Evoformer consists of 48 neural network blocks. Again, I'll be talking about what exactly that block does in a little bit. But after it— the representations are updated and come out of this core part of the network, we then feed those richer and updated representations in what we call the structure module. The structure module really takes these abstract representations and predicts sort of a concrete realization in 3D space of the protein and the side chains of, of the protein. And really, that module consists of 8 blocks. Again, I'll give more details what exactly the structure module consists of in the slides to follow. Another thing that we found very helpful was a concept we call recycling, where, as the name might suggest, we take the representations that the network found at the very end and recycle them through the network again, which is to say we essentially just additively add them to the inputs again and perform that up to 3 times. And the network weights that you use during these iterations, they are the same. So really it's just input recycling, and we've found this to consistently increase performance. So that was a nice little extra trick that gave extra performance. There are 2 other outputs to the neural network that we predict. Both of them relate to confidence estimates, which are really important to understand where and when to trust AlphaFold in the predictions that it makes. That has been found to be really helpful in biological applications already. So one of them is a per-residue confidence estimate that we call PLDDT, and then there's also a pairwise confidence estimate that we call the predicted aligned error. Why there's two and what exactly they are, again, I'll talk to that in a little bit. Okay, so you might be curious which part of sort of this entire machinery mattered, and in a nutshell, we can say all of it. On the right-hand side, you see an ablation study, a figure that we've put in our publication where we knocked out essentially individual components that I've just described. And what you find is if you retrain without these components, each time it resulted in a worse performance. So that really goes to show that all of the individual ideas and pieces that we've put in conspire in a way, if you will, to the final performance. So it's really not a single improvement that is dominant in that case. Okay, as promised, some more insight on the fundamental building block in Evoformer that is at the core of this network. So again, we have an MSA representation depicted in orange here, and then we have a pairwise representation depicted in blue. And like I said, they update one another in an interleaved way, and this works as follows. So on the MSA representation, we have self-attention mechanisms, and in those row-wise self-attention pieces here, we bias the attention affinity matrix essentially using the pairwise stack, and that really goes into the update of the MSA stack. Now, once we've updated that stack, we then go ahead and outer product the MSA representation and then add a projected version of that outer product to the path stack. And that's really this, uh, yeah, back and forth update between the two different stacks. After that, we do a sort of special update on the pairwise representation that we call triangular attention. And it's one of those ways with which we've put geometric insights into the network. On the next slide or two, I'll explain what that piece is. So triangular attention really is, as I said, a way of putting notions of geometry, Euclidean geometry, into the network. If you think of just general geometry, 3 points A, B, and C, if you know the distance between point A and B and the distance between point B and C, that puts a really, really strong constraint on the distance AC, like just by means of the triangle inequality and so forth. And that's something that's not normally enforced in a network like that. And really, we hope that the pair input embedding encodes these spatial relationships and infers those spatial relationships, and therefore that's the place to put such knowledge in, such biases. And really, what we attempted to do was therefore to have the update for the pair (a, c) depend on the representations of the pair (b, c) and (a, b). That's sort of shown on the right-hand side here, where in the pairwise stack, each pixel, if you will, each position encodes a pair. And therefore, if you want to update the green pixel in this space, this edge in the structure shown underneath, then you should do that on all possible loops that you can form using the red and the yellow edges. So we really treated this sort of as a graph inference problem where the graph isn't known but has to be inferred, and the edges of these graphs are the pairs of residues. Like I just said, the update for the green edge therefore should involve all cycles involving the possible edges that it is part of— sorry, all cycles that it is possibly a part of. And in practice, what we've done is we summed over all of the contributions of the column that it's in and all of the contributions of the row that it's in. And really, this puts in this transitivity inductive bias that encodes some prior knowledge about Euclidean space. And also in the ablations, we found that to, to really help the network reason about this space. All right, um, now some more, um, color to the structure module. So first of all, I, I want to point out that that was the first time, to our knowledge, that someone had attempted to do end-to-end folding, by which we mean really directly predict coordinates. Before, people only predicted pairwise distances and then did some gradient descent on that estimate to come up with concrete structures. The way we've done that is to parameterize the protein backbone, which again is this chain where the amino acids are connected up at, as a gas of 3D rigid bodies. So we didn't— we in fact didn't treat it as a chain, but let the network learn that itself. We cut up the chain that is shown in this sort of cartoon on the left into little triangles, the red pieces, and then let the network learn that these triangles need to be connected up. And the triangles themselves, they are essentially parameterized as a translation and a rotation. So it's a rigid body frame that the network predicts given the representations that it is fed from the Evoformer trunk. And, uh, yeah, The block itself, given this parametrization, is a 3D equivariant transformer architecture that updates these rigid bodies and also predicts the side chains parametrized as chi angles with respect to the backbone frames. And on the right-hand side in this GIF, you can see that the network already in iteration 1 has a very good guess of what the structure is, and then over the 7 iterations that follow, it really refines that structure estimate and arrives at a final prediction. Um, cool. Another piece that was very helpful and I wanted to point out is what is called noisy student distillation. So again, I want to highlight that that's not something we invented, but we found useful and took from the, from the literature. Uh, but I think it's a very interesting piece, uh, and therefore wanted to include it. So AlphaFold really makes use of, um, unlabeled sequences. Um, and the, the reason for that is that what we can train on is the experimental structures that are sort of known to humanity. And really, in terms of machine learning, it's not all that much. It's about 150,000 structures. That by itself, um, doesn't get you to the final performance that we've achieved with AlphaFold. But instead, what we've done is, given an initial version of AlphaFold trained on these 150,000 structures, we predicted the structures for sequences that were unlabeled, by which we mean didn't have a known experimental structure. And by that procedure, we produced millions of predicted structures from this initial model. And then we trained that model again, but on its own predictions. And that might seem sort of strange and circular at first, because really it's sort of bootstrapping yourself out of your own to a higher performance. But yeah, it really was something that helped. And the thing that we feel like is the mechanism at heart is sort of a regularization of the model by being forced to being consistent with an earlier version of itself. All right, so that was the network architecture. And with this, I'd like to move on to sort of talking about how AlphaFold seems to understand proteins. So one interesting piece to highlight is that protein structure prediction is typically underspecified. That's because proteins take on their structure not in isolation, but sort of given, you know, all the context that's around them— the waters, small molecule small molecule ligands that might bind to it, sugars, DNA pieces that might bind to it, or also just the experimental conditions like temperature, pH, all these kind of things. So in that sense, protein structure prediction is underspecified because the network is not told what sort of context it should be using to make its current prediction. And for that reason, our network seems to implicitly model this missing context. Simply by virtue of being forced to do that on the training sets. And on the bottom of this slide, I'm showcasing two interesting cases where it does that. So on the left, you can see a GIF where both the ground truth in green and the AlphaFold prediction in blue rotate around this gray sphere. And this gray sphere is a zinc atom. Now, the zinc atom is only present in the experimental structure, So these, these, the sidechains that you can see there, these sort of circular, these, these might not be— whoops, I didn't mean to do that. Okay. Never attempt that again. Oops. Great. Let me quickly flick through that. Okay, so what I was trying to explain is that these three side chains coordinate this zinc atom, meaning they're really bound to the zinc atom, and the side chains only take on this specific, uh, conformation because the zinc atom is there. And you can see that AlphaFold in blue predicts exactly that side chain conformation as if the zinc atom was there, but it doesn't know about it. So we think it, you know, must have seen similar training cases and implicitly models that a zinc atom is present there without ever being told that it is. And then on the lower right-hand side, you can see the example of a trimer. A trimer is a protein consisting of 3 protein chains, so essentially 3 proteins forming a larger protein, if you will. And in this case, this trimer consists of 3 copies of the same base chain.. And our model as published and as used at CASP only models single chains. So in this case, we asked AlphaFold to fold this chain, and it predicted the structure as if this chain was forming this trimer. So on the left-hand side, we're triplicating this chain in a C3 symmetry, and you see that it forms the same structure as the experimental structure of the trimer. So AlphaFold, just given the sequence, must have realized, if you want to anthropomorphize here, that this is a trimer. And that's sort of what we mean by it needs to— it seems to sort of implicitly model the missing biological context. All right, another thing that's quite informative is interrogating the network by freezing the final architecture and weights and then retraining small probing heads at various different depths of the network. And then these, these heads are trained to predict the structure. And what that tells you is sort of how rich and how good the representation is at any given depth in the network. And that's what's shown in this GIF on the right-hand side that corresponds to this SARS-CoV-2 structure that I've showed earlier that was part of CASP. And you can see over roughly 200 layers, the network sort of blows up the structure, makes these very coherent moves, reorients, reshuffles different parts, changes its prediction, and finally arrives at what is a very good prediction, as you can see on the left. So it's quite interesting, and we found that a very nice way of interrogating the network. Same thing for completion for the large RNA polymerase. So again, over like 200 layers, the model makes its prediction. It blows up in space very early on, and then it makes these well-coordinated, coherent moves almost. It moves around individual domains, as you can see, until it's happy, if you will, and again predicts a very good structure. And yeah, importantly, I'd like to point out, you know, the network is not trained to do that. The network weights were frozen, and that's just how it seems to operate in that space. All right, so next section is on how to interpret the predictions that you get from AlphaFold. So one of the confidence estimates that AlphaFold produces is called predicted LDDT. LDDT stands for Local Difference Distance Test. What exactly it is doesn't really matter, but essentially it's a local metric of structural agreement where higher is better. And what we've done is we trained a small head that predicts the LDDT error that the network makes during training, and then given new cases, we can look at the predicted error that the network, you know, says it makes for this new sequence. And that's already been, you know, considered extremely useful by biologists because it really tells them when and when not to trust the structures. So I wanted to give a brief intuition. When the PLDT is below 50, we think that typically these backbone predictions aren't really to be trusted. The slight caveat is that possibly these PLDDT less than 50 predictions actually correspond to real disorder. So proteins actually can have disorder and at least partially never take on a well-defined structure. And so we've by now found good correlation with actual disorder. So PLDDT less than 50 means don't trust the structure, but it also means maybe this is really not taking on any structure. So this is quite interesting. PLDDT of larger than 70 typically means that you can work with the backbone predictions, but you may not want to, you know, trust the sidechains in that area. So it's a good prediction if it's above 70, but maybe the sidechains wouldn't be correct. And then on the right-hand side, you can see a different prediction where large parts are colored in the darker blue corresponding to a PLDDT of above 90. And there we think it's really already reasonable to investigate the side chains. In that sort of domain, AlphaFold often reaches atomic accuracy, so better than 1 angstrom all-atom accuracy. And that's, that's really good where, you know, the side chains become usable. You might be able to do some docking of small molecules and that kind of thing. All right, now, so predicted LDDT has an important pitfall. That is that, as I said, it does a local assessment of the accuracy, but it doesn't say anything about relative errors. So in the protein shown in the center of this slide, you see a protein that's composed of essentially two subunits that are called, called domains., and in this case it's connected up by a small linker. And these domains, really, they are rigid themselves, but they might flop around against each other. So you could in this case very likely trust the individual domains, but not how they are situated in space relative to one another. And to basically have that come across and have a quantitative assessment of this relative kind of error, we came up with a second error estimate that we call the predicted aligned error. And the way this works is suppose residue Y was aligned to the true structure and we measured the position error of residue X, then this is the error that we're trying to predict. So again, this is shown for the same protein consisting of these two domains, and the predicted aligned error is really this pairwise matrix shown on the right-hand side. Maybe I should re-explain. I think I didn't quite do it justice. So in this matrix, given any position ij, the error that you see in this matrix is the error for residue j if you are aligned to the residue reference frame of residue i. And in this case, the error is really very useful and indicative because you can see the two dark green squares on diagonal that correspond to the two domains of these proteins. There, AlphaFold essentially tells you, you can trust these domains. The relative error within this square is very low. They are rigid, basically, and that's where the prediction is really good. But then off diagonal, the two white squares, that's where, in this case, AlphaFold essentially tells you, don't trust me how I positioned these two domains with respect to one another. And again, yeah, very useful information for, say, biologists that want to work with these structures. Okay, let me briefly talk about how you can access AlphaFold. So one thing that we've also published is the AlphaFold protein structure database. It essentially is a website that was developed and hosted by EMBL-EBI together with DeepMind. Initially, it was hosting the AlphaFold predictions for 20 model organisms as well as the whole human proteome, and that was about 350,000 structures. Since January this year, it's been expanded to another 27 model organisms comprising another 190,000 structures. And within this year, we're planning to release the predicted structures for a sequence dataset that's called UNREF90, and there'll be another 100 million, in this case, structures where this comes close to all sequences that are, you know, known to humanity at this point. So yeah, I encourage you to check out the structure database, and of course you can, you can bulk download the structures. You also get the uncertainty estimates that I've just talked about, and yeah, really can sort of use structures that were previously not available, including the whole human proteome. Another way of accessing AlphaFold is through the AlphaFold Colab. A Colab is essentially a website hosting a pre-written Python program, in this case, of course, AlphaFold. It runs in the cloud, in this case the Google Cloud. Typically it's backed with a Google GPU, and so it's really almost plug and play. You enter an amino acid sequence and just, yeah, go through these steps in this website, and, you know, after some time which is, you know, length dependent, how long AlphaFold takes to make the prediction. You'll get a predicted structure and again also these confidence estimates that you are then free to use. And the third way is we've also fully open sourced the entire code and weights, so you can just download AlphaFold and yeah, essentially by now do whatever you want with it. There's also a Docker container to make it even easier, and I encourage you to check out our GitHub page if you're interested in running it in that way. Um, all right, um, now let's get into sort of how the biology community has already made use of AlphaFold. So just generally speaking, uh, we as a team have been very keenly following Twitter, uh, as probably many of you are aware. That is where science these days happens, uh, at full pace, at the fastest, um, you get very immediate feedback. And it was really interesting to see, you know, what the community thought and how it immediately built on AlphaFold and extended it. And, and was, was really interesting for us to, to follow along. So one of the things that happened quite early on was AlphaFold's been integrated into existing tools, and that's been, you know, great because it's made using it for the community, the structural biology community for instance, even easier if they can just use it in the existing tools that they're used to using. Then people have expanded upon our Colab, essentially come up with different versions that can do, you know, slightly orthogonal things. And that's also been super, super nice to see and has further expanded the reach of this work. Another thing that's interesting to highlight is that it's also accelerated experimental structure determination. Now again, this might seem sort of paradoxical almost, because we set out to, you know, predict structures, not help with experimentally solving them. But the reason why our predicted structures can actually help with experimentally solving them is that the data that you get back from these crystallography experiments, for instance, that underspecified— you need an initial model to fit that data. And that's where AlphaFold was able to help with some particularly tricky, tricky cases already where there was no good initial model. And really it's helped, uh, yeah, solve some, in some cases, uh, structures that people had experimental data for since 10 years, but they just weren't able to get the structure from those measurements. And so yeah, all this was very interesting and cool to see. Um, yeah, I already mentioned that proteins can have intrinsic disorder, so never take on a well-defined structure. And without explicitly being trained to that, trained for that, AlphaFold seems to be at least on par with existing, you know, best-in-class methods for disorder prediction. So that's what the externals have found. It basically— just using PLDDT, you can, uh, yeah, very well determine, uh, unstructured regions. And then, um, given, you know, the AlphaFold database with the currently hundreds and thousands of structures, people have already started to do large-scale data analyses of the things that are in, in that structured database, um, you know, looking at various properties, trying to extract information from that database, which I think there'll be much more to come of that sort of work as well. And it's super cool to see. Lastly, maybe one of the coolest works personally that I've personally seen is this piece where people have used AlphaFold predictions to sort of model this huge protein complex consisting of over 500 proteins. So this is a nuclear pore that is, you know, bound, or is a pore in a cell membrane. And this is, to my knowledge, the largest protein complex that was ever experimentally resolved. And AlphaFold was used in this case to puzzle together how exactly the proteins fit this pore. Essentially, the the data that people took experimentally is too coarse to solve the puzzle, if you want to stay in that metaphor. And that's where AlphaFold came in and was used to do this integrative modeling for this huge complex, which is really a nice piece of work. All right, and then sort of the last point of the talk is about briefly how AlphaFold can be used to predict protein interactions and protein complexes. So very interestingly, after we've released the model, only like 2 or 3 weeks in, people started to realize— externals— that you can hack AlphaFold to predict the structure for 2 chains. So AlphaFold was only trained to predict the structure for a single chain, but proteins are often in contact with other proteins forming protein complexes, as I had already alluded to. Now, um, people basically just hacked it in ways that we hadn't anticipated. So they found tricks to make that work. For instance, They, in the central piece colored dark, what they did there was use two chains and put glycine linkers between the two chains, which are typically— which are amino acids that form typically floppy loops, so they don't really contribute to structure. So they basically hacked the system to think that these two sequences are one by putting in such a loopy linker, and the model in that case perfectly predicted the structure of both in contact without ever being trained for that. And then another hack that people found was basically we tell the model sort of the absolute position that the residue takes on in the sequence. And if you tell AlphaFold that two chunks of a sequence are sort of further apart given that position embedding, it can again fold them as if it was one chain., but it effectively treats them as two individual chains. And that's what people in that case made use of to fold two chains and sort of making AlphaFold think it's a single chain, if you want to put it that way. So that was super interesting to see. So of course, in the meantime, we had already worked on a system that was dedicated to that problem and sort of had various architectural and training alterations. It was specifically trained for the multichain case, and that has since also been published, and it beats pre-existing methods here in terms of this DocQ score, which is used in the community. And it also beats the hacked AlphaFold version by quite a margin. And again, it's also available open source. For instance, the Colab has been extended, so if you want to predict— sorry, predict the structure for more than one sequence, that's also possible in the Colab by now. Okay, and then briefly, um, let me briefly touch on future work. So in structural biology in general, there is of course lots of exciting work ahead for the field. AlphaFold, you know, doesn't solve everything in that regard at all. There's much to be done, one of them being the prediction of structures for complex complexes. Again, AlphaFold Multimer is, is, um, existing proposal for that, but I think it still has some, uh, some miles to go to be fully reliable. Then disorder is one such problem that people are interested in and isn't solved. Uh, then there's conformational changes where, you know, proteins are really dynamic beasts actually, but here we've treated them as if they weren't. So that's one thing that people are interested in. You want to understand how proteins, uh, you know, dynamically change shape essentially. And then another thing that's super interesting, um, uh, single mutations or point mutations as they are called. So making changes to the sequence and then seeing what effect it has on the structure. Unfortunately, currently in the current version of AlphaFold, that isn't reliable. So we know there's work to be done to correctly model point mutations, for instance. And then as I said, we're super excited to see what others build on top of the AlphaFold database. And there's probably lots to, to come out of analyzing that very closely as well. And just sort of generally speaking, as a general musing for AI for science, I think there's, you know, great potential for building these highly optimized machine learning systems for scientific problems. Um, you know, AlphaFold is, is only one of many that are already out there and many to come. Um, I think we'll see many breakthroughs using that kind of dedicated machine learning system.. And in that vein, AI can really, I think, help see scientists further make sense of large amounts of data that you couldn't by eye or in a handcrafted way. And that's really where the large benefit of that kind of system comes to play. All right, with this, I'll close my talk. Thank you for your kind attention. And of course, want to point out that this work obviously wasn't just mine. I was part of a team of 15, and then there's a much wider team within DeepMind and, and so on that have contributed to that and, and worked on that, and very important to acknowledge them. So thanks everyone, and I think we'll move on to questions and answers.