DeepMind AlphaFold: The Nobel Prize Winner! 🧬

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Oh my goodness. This work is history in the making. Today we are going to have a look at AlphaFold, perhaps one of the most important papers of the last few years, and you will see that nothing that came before even comes close to it and that it truly is a gift to humanity. So what is AlphaFold? AlphaFold is an AI that is capable of solving protein structure prediction, which we will refer to as protein folding. Ok, but what is a protein and why does it need folding? A protein is a string of amino acids. These are the building blocks of life. This is what goes in, which in reality has a 3D structure and that is protein folding. Letters go in and a 3D object comes out. This is hard. How hard exactly? Well, let's Compare it to DeepMind's amazing previous projects and we'll see that none of these projects even come close in difficulty. For instance, DeepMind's previous AI learned to play chess. Now why does this matter as we already have Deep Blue, which is a chess computer that can play at the very least as well as Kasparov did, and it was built in 1995. So why is chess interesting? Well, the space of possible moves is huge, and deep blue in 1995 was not an AI in the strictest sense, but, but a handcrafted technique. This means that it can play chess and that's it. One algorithm, one game. If you wanted to play a different game, you write a different algorithm. And yes, that is the key difference. DeepMind's chess AI is a general learning algorithm that can learn many games. For instance, Japanese chess, Shogi two, one algorithm, many games. And yes, chess is hard, but these days the AI can manage. Then Go is the next level. This is not just hard, it is really hard. The space of possible moves is significantly bigger and we can't just evaluate all the long term effects of our moves. It is even more hopeless than chess. And, and that's often why people say that this game requires some sort of intuition to play. But DeepMind's AI solved that too, and beat the world champion Go player 4 to 1 in a huge media event. The AI can still manage. Now get this, if chess is hard and Go is very hard, then protein folding is sinfully difficult. Once again, a string of text encoding the amino acids go in and a 3D structure comes out. Why is this hard? Why not just try every possible 3D structure and see what sticks? Well, not quite. The search space for this problem is still stupendously Large. Perhaps not as big as playing a continuous strategy game like Starcraft ii, but the search here is much less forgiving. Also, we don't have access to a perfect scoring function, so it is very difficult to define what exactly should be learned. You see, in a strategy game, a win is a win. But for proteins, nature doesn't really tell us what it is up to when creating these structures. Thus DeepMind did very well in chess and Go and Starcraft 2. And challenging as they are, they are not even close to being as challenging as protein folding. Not even close. To demonstrate that. Look, this is CASP. I've heard DeepMind CEO Demis Hassabis call it the Olympics of protein folding. If you look at how teams of scientists prepare for this event, you will probably agree that yes, this is indeed the Olympics of protein folding. At about a score of 90, we can think of protein folding as a mostly solved problem. But no need to worry about definitions though. Look, we are not even close to 90. And it gets even worse. Look, this GDT score means the global distance test. This is a measure of similarity between the predicted and the real protein structure. And wait a second. What? The results are not only not too good, but they appear to get worse over time. Is that true? What is going on here? Well, there is an explanation. The competition gets a little harder over time. So even flat results mean that there is a little improvement over time. And now hold on to your papers and. And let's look at the results from DeepMind's AI based solution, AlphaFold. Wow. Now we're talking. Look at that. The competition gets harder and it is not only flat, but can that really be. It is even better than the previous methods. But we are not done here. No, no, not even close. If you have been holding onto your papers so far now, now squeeze that paper because what you see here is old news. Only two years later, AlphaFold2 appeared. And just look at that. It came in guns blazing. So much so that the result is. I can't believe it. It is around the 90 mark. My goodness, that is history in the making. Yes, this is the place on the Internet where we get unreasonably excited by a large blue bar. Welcome to Two Minute Papers. But what does this really mean? Well, in absolute terms, AlphaFold2 is considered to be about three times better than previous solutions. And all that in just two years. That is a miracle right in front of our eyes. Now let's pop the hood and see what is inside this AI. And look at all these elements in the system that make this happen. So where do we even start? Which of these is the most important? What is the key? Well, everything and nothing. I will explain this in a moment. That does not sound very enlightening. So what is going on? Well, DeepMind ran a detailed ablation study on what mattered and the result is the everything mattered. Look, with few exceptions, every part adds its own little piece to the final result. But none of these techniques are a silver bullet. But to understand a bit more about what is going on here, let's look at three things. One, AlphaFold two is an end to end network that can perform iterative refinement. What do these mean? What this means is that everything needed to solve the task is learned by the network and that it starts out from a rough initial guess and then it gradually improves it. You see this process here and it truly is a sight to behold. Two, it uses an attention based model. What does that mean? Well, look, this is a convolutional neural network. This is wired in a way that information flows to neighboring neurons. This is great for image recognition, because usually the required information is located nearby. For instance, let's imagine that we wish to train a neural network that can recognize a dog. What do we need to look at? Well, floppy ears, a black snout, fur. Ok, we're good. We can conclude that we have a dog here. Now, have you noticed? Yes, all of this information is located nearby. Therefore, a convolutional neural network is expected to do really well at that. However, check this out. This is a transformer, which is an attention based model. Here information does not flow between neighbors. No sir. Here information flows everywhere. This has spontaneous connections that are great for almost anything. If we can use them well. For instance, when reading a book, if we are at page 100, we might need to recall some information from page one. Transformers are excellent for tasks like that. They are still quite new, just a few years old, and are already making breakthroughs. So why use them for protein folding? Well, things that are 200amino acids apart in the text description can still be right next to each other in the 3D space. Yes, now we know that for that we need attention networks. For instance, a transformer. These are seeing a great deal of use these days. For instance, Tesla also uses them for training their self driving cars. Yes, so these things mattered, but so many other things did too. Now I mentioned that the key is everything and nothing. What does that mean? Well, look here, apart from a couple examples, there is no silver bullet here. Every single one of these improvements bumped the score a little Bit, but all of them are needed for the breakthrough. Now, one of the important elements is also adding physics knowledge. How do you do that? Well, typically the answer is that you don't. You see, when we design a handcrafted technique, we write the knowledge into an algorithm by hand. For instance, if in chess, there are a bunch of well known openings for the algorithm to consider for protein folding, we can tell the algorithm that if you see this structure, it typically bends this way. Or we can also show it common protein templates, kind of like openings for chess. We can add all this valuable expertise to a handcrafted technique. Now, we noted that scientists at DeepMind decided to use an end to end learning system. I would like to unpack that for a moment because this design decision is not trivial at all. In fact, in a moment, I bet you will think that it's flat out counterintuitive. Let me explain. If we are a physics simulation researcher and we have a physics simulation program, we take our physics knowledge and write a computer program to make use of that knowledge. For instance, here you see this being used to great effect. So much so that what you see here is not reality, but a physics simulation, all handcrafted. Clearly, using this concept, we can see that human ingenuity goes very far and we can write super powerful programs, or we can do end to end learning, where surprisingly, we don't write our knowledge into the algorithm at all. We give it training data instead and let the AI build up its own knowledge base from that. And AlphaFold is an end to end learning project. So almost everything is learned. Almost. And one of the great challenges of this project was to infuse the AI into with physics knowledge without impacting the learning. That is super hard. So training, huh? How long does this take? Well, get this. DeepMind can train this incredible folding AI in as little as two weeks. Hmm, why is two weeks so little? Well, after this step is done, the AI can be given a new input and we'll be able to create this 3D structure in about a minute. And we can then reuse this trained neural network for as long as we wish. So this is a lot of trouble to fold these proteins. So what is all this good for? The list of applications is very impressive. I'll give you just a small subset of them that I really liked. For instance, it helps us better understand the human body, or create better medicine against malaria and many other diseases, develop more healthy food, or develop enzymes to break down plastic waste. And that is just the start. Now you're probably asking you Keep saying that this is a gift to humanity. So why is it a gift to humanity? Well, here comes the best part. A little after publishing the paper, DeepMind made these 3D structure predictions available free for everyone. For instance, they have made their human protein predictions public. Beyond that, they already have made their predictions public for yeast, important pathogens, crop species and more. And thus I have already seen follow up works on how to use this for developing new drugs. What a time to be alive. Now note that this is but one step in a thousand step journey, but one important step nonetheless. And I would like to send huge congratulations to DeepMind. Something like this costs a ton to develop and note that it is not easy or maybe not even possible to immediately make a product out of this and monetize it. This truly is a gift to humanity. And a project like this can only emerge from proper long term thinking that focuses on what matters in the long term, not just thinking about what is right now. Bravo. Now of course, not even AlphaFold2 is perfect. For instance, it's not always very confident about its own solutions and it also performs poorly in antibody interactions. Both of these are subject to intense scrutiny and follow up papers are already appearing in these directions. Now one last thing. Why does this video exist? I got a lot of questions from you asking why I made no video on AlphaFold. Well, protein folding is a highly multidisciplinary problem which beyond machine learning, requires tons of knowledge in biology, physics and engineering. Thus my answer was that I don't feel qualified to speak about this project, so I better not. However, something has changed. What has changed? Well, now I had the help of someone who is very qualified, as qualified as it gets, because it is the one and only John Jumper, the first author of the paper, who kindly agreed to review the contents of this video to make sure that I did not mess up too badly. Thus, I would like to send a big thank you to John, his team and DeepMind for creating AlphaFold and helping this video come into existence. It came late so we missed out on a ton of views, but that doesn't matter. What matters is that you get an easy to understand and accurate description of AlphaFold. Thank you so much for your patience. This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights and weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning and even hyperparameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link wnb me paperintro or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back. Thanks for watching and for your generous support and I'll see you next time.