Scientist Stories: Jian Tang, Generative AI in Biology

Thank you to everyone around the world for joining us for this week's Immunome Lab Meeting. My name is Kristen Abood. I am a science editor at the Human Immunome Project, and I will be your moderator today. Chances are you've all heard of or even used the AI-backed chatbot ChatGPT since it launched late last year. In fact, I tried using it to write the introduction to this lab meeting, and unfortunately, each time I tried, it was at capacity. And instead, it offered me this guided meditation on the status of ChatGPT. Trust that the right time for you to try ChatGPT will come. In the meantime, continue to focus on your breath and stay present in the moment. Fortunately for us, our speaker today is not at capacity or offering us a guided meditation in lieu of his talk, which will be about the equivalent of ChatGPT in biology. And how generative AI can be applied to antibody and protein design. But before we get started with today's presentation, I just want to share with you an upcoming event to honor the 2022 Michelson Prize winners, which will take place as a virtual panel in collaboration with the Keystone Symposium on March 24th at 11:00 AM Eastern time. The Human Immunome Project and the Michelson Medical Research Foundation have awarded 4 outstanding early career scientists with $150,000 grants to pursue their work to advance human immunology, vaccine discovery, and immunotherapy research. This year's winners are an impressive group, and we invite all of you to join the virtual awards ceremony. You can register by clicking on the QR code on this slide or by visiting our website for more information. Please also note that the application portal for the 2023 Michelson Prizes will open on April 3rd. Just one more note before we begin. At the request of our speaker, we ask that you not reproduce or disseminate the data presented in today's webinar in any way. We thank you so much for your understanding. And now it is my great pleasure to introduce today's speaker. Dr. Jian Tong is an associate professor at the Mila-Québec AI Institute and the founder and CEO of BioGeometry, an AI startup focusing on antibody discovery. Dr. Tong's diverse research interests include graph representation learning, graph neural networks, geometric deep learning, deep generative models, knowledge graphs, and drug discovery. Recently, his group released an open-source machine learning package called TorchDrug and TorchProtein, which aim to make AI drug discovery software and libraries freely available. During today's presentation, please send me your questions using the Q&A function in Zoom. I will ask our speaker a broad selection of your questions after the presentation, and then we'll have some time for discussion. With that, it is my pleasure to now turn it over to Dr. Tong.

Thanks, Kristen, for the introduction and also the invitation. It's a great pleasure to present here. So let me try to open up my camera, or maybe Kristen, you probably need to give the— okay. I can start sharing the screen. I still cannot, like, start my video.

Okay, let's see if we can. I'm going to Try that.

Oh, it works.

Okay, great.

Okay, so thanks so much for the invitation and also the great opportunity in speaking here. As Kristen said, like, now ChatGPT is really like, is a really popular topic, right? Everyone is like talking about ChatGPT and how can we use ChatGPT in our life, in our study, and in our research. But what will be the corresponding ChatGPT in biology? Can we use the technology developed in ChatGPT for many important problems in biology? So before the presentation, I want to claim that I'm not a biologist. I'm actually a computer scientist. But actually, I'm really fascinated by many problems in biology and also how artificial intelligence can be used to solve important problems in biology. I can start with some big picture. Before that, maybe I also try to disclose, besides being an associate professor at the Mila-Quebec AI Institute, I also serve as a scientific advisory board in the UK startup, Ration Therapeutics, which focused on using AI for understanding the causal relationship between genes and disease. And also founder of the startup BioGeometry, which focused on using geometric deep learning and also generative AI for antibody design. Okay. So, I'll just start with some— A big picture. I think for people working on biology now, pretty much I think we are at the best time because we are now seeing a revolution in both AI and biology. So on the artificial intelligence side, I think in the past 10 years, AI really has made huge, huge progress, right? So starting from its first revolution back to 2012, so we see like a new network is able to recognize the objects in the image where we are. So basically the new network is able to get the best performance on the ImageNet dataset. Afterwards, there are many important breakthroughs for AI. For example, back to 2016, the AlphaGo system developed by Google's DeepMind, they beat the best human Go players. The essential algorithm developed in AlphaGo is a reinforcement learning algorithm for the game of Go. And in recent 2 years, we've seen really very exciting progress of generative models, especially large language models, for example, GPT-3 and also ChatGPT. So we have been talking about recently. And also diffusion models, which basically we can use diffusion models to generate images. So all these models belong to generative models, either generating images or generating text. So another type of methods which have made huge progress is really the graph machine learning, graph neural networks, graph machine learning, or geometric deep learning methods. So maybe some of you are not quite familiar with this topic, but one very successful example which I'm pretty sure you are pretty familiar with is AlphaFold 2 system, also developed by Google's DeepMind. So the AlphaFold 2 system is essentially built on top of graph machine learning or geometric deep learning techniques. Because for AlphaFold 2, the very essential idea is to use deep learning to model the relationship between the residues in the spatial space. And once we are able to model the relationship between the residues, so we can easily predict their structures in the 3D space. So you see, so we really have a lot of excitement for AI in the in the past 10 years. Okay, so that's one, one side of revolution which may be not quite familiar with you. Okay, but then on the other side, biology, which are relatively like not new to us, but I believe you're quite familiar with the, the exciting techniques in biology. Okay, so in biology, I think for me, like, I think the most exciting technique is really, really the, the high-stoop DNA sequencing, editing, and also synthesize, right? So nowadays we are able to do DNA sequencing and synthesis in a very cheap way and also high-throughput way. And why that makes sense? Because it makes a huge difference. Because with the high-throughput DNA sequencing and also synthesis, we are basically— okay, we can sequence a lot of proteins like DNAs. So we can understand the functions of the cells. So we can generate a lot of data. So with the capability of DNA synthesis, we can create new DNAs, new proteins. So that's actually helped us understand the functions of like DNAs and also the cells. Okay. So this is one type of like, I think, a revolutionary technique in biology. So another type of like important technique in biology, I think that's also starting to make also huge progress recently is cryo-EM technique. So with recent techniques, now we are able to like solve the protein structures much, much faster. So which means that, okay, so we are able to solve have more and more protein structures, which really help us understand the protein structures much faster and also better. So we see two kinds of revolution in both AI and biology. So we can go into a little bit details. For example, going back to AI side, I want to specifically emphasize the exciting progress that has been made in generative AI. So what is the essential idea of generative AI? So generally, AI is like this. Okay, so first we have a huge amount of like dataset, let's say, on the internet. So we have a lot of like text data and also image data, maybe also speech data, right? So what generative AI do is essentially we're going to train a model which we call like generative model. Okay, so we train a generative model based on the huge amount of data, the text data, image data, and speech data. So once we train that model, then the model has the capability of generation, or we can generate new data from this model. So we can either generate text, images, or speech from this model. So that's called generative models, or what we call generative AI now. So I can give you a few examples. So one particular example, which I believe Many of you also have heard about this, which is large language models, which is used for text generation. So one very well-known example is GPT-3, which was developed by OpenAI. So GPT-3 basically allows you to generate new text. So let me give you one example. Let's say, OK, you can give a sentence, a starting sentence. Then the model is able to help you to complete the rest of the paragraph or the document, depending on how many new sentences or new words you want to generate. So that's one example of doing text generation based on GPT-3, which you train pretty much on the entire text corpus on the internet. So that's one type of generation, text generation. So besides generating text, of course, we can also generate new images. So specifically, we can actually generate new images according to a text. So basically, OK. Give a text description. So we want to generate some images whose semantics aligns well with the input given text. So there are many very well-known models in this space as well. For example, like DALL·E 2, also developed by OpenAI, which is essentially an image generation model based on text. And also another very well-known model is the Stable Diffusion model, which was released also last year. So here, let me give you one example. So this is essentially like an image generation model based on— image generation example based on DALL·E 2. So here, you can input a text prompt, photo of a panda surfing. And then the model will generate an image like this, which pretty much like the semantic of this image pretty much aligns well or depends on the input text description. So that's from text to image generation. Okay. So nowadays, of course, as we just said, like everyone is talking about like ChatGPT, right? So what is ChatGPT? So ChatGPT is essentially a large language model and also fine-tuned for conversation. So from the point of view of conversation, so ChatGPT is essentially a conversation system, right? So I'm not sure how many of you have played around with ChatGPT. So basically, you can make conversations with computers, with algorithms. You can input a sentence, and then ChatGPT will generate the response for you. So remember that all the responses by ChatGPT, so these sentences are not copied from elsewhere. So they're actually generated by the algorithm. So this response may be similar to some of the text we have on the internet, but they're not 100% overlap with existing ones. So in other words, so these answers, these responses, usually they are new. So you can have multiple rounds of conversation with ChatGPT. So that's ChatGPT. So maybe some of you are interested in, OK, how in practice, how is ChatGPT trained? So here, let me show you some idea. So ChatGPT, so first, we have very large language models. So ChatGPT is actually trained based on the text data, pretty much all the text on the internet. So this one type of data. But besides text data, We also have the code data, the code, basically the programming language, the codes written by the programmers. And that's why actually ChatGPT is able to understand not only the test data, but also is able to do debugging because it's also able to understand codes, programs. So ChatGPT is pre-trained with these two types of data using large language models. They call it like GPT-3.5, which is, I think, is a bit more advanced than GPT-3 they had in the past. So that's a pretrained model. So remember that here, this is just a pretrained model. So you can do text generation. You can generate codes. You can generate a text. But remember that this pretrained model is not tuned for conversation, which means maybe it can be used for conversation, but it's not optimal for conversation. And that's why. They first fine-tune this model for conversation purpose. So what they do is first, they also do some supervised fine-tuning. So what they did is, OK, they first collect a lot of label data. So the label data is like this. In some conversation context, they basically ask humans— not ask the algorithms to generate the response, but basically ask humans to write the answers. So they will collect many such kinds of data. The conversation context, and then human response. So we have many, many this kind of paired data or labeled data. Based on this data, so we can fine-tune the model. Okay, so we fine-tune the model based on supervised learning, which fine-tunes the model based on this labeled data. Okay, and this second step. And further, so once we have this model, we can actually further let this model to interact with humans, okay? And then by interacting with humans, human can further provide rewards or informative signals to further fine-tune the model. So, after these 3 steps, okay, then we are— so, then we got a model which is fine-tuned for conversation purpose, okay? So, that's the high-level idea. Of like developing or training ChatGPT. Okay, so that's the high-level idea of generative AI for different applications. Okay, so that's one side. So another side, as we said, like nowadays in biology, okay, so we have very exciting breakthroughs in DNA sequencing, DNA synthesis, and also cryo-EM. And then As a consequence, the good thing is like, okay, now the number of data in barrage is increasing really, really, I think, at a fast speed. I think the increasing rate actually is larger than the Moore's law. Okay. So let me give you a few examples. For example, because of like, thanks to the DNA sequencing, nowadays we are really able to do a lot of like protein sequencing and also BCR sequencing, right? So we have a huge dataset of protein sequence, like UniProt, and also antibody sequence, like OAS dataset. So for this sequence, they are unlabeled, which means we do not have other information, for example, the functions of this sequence. So we call this unlabeled sequence. So another type of data we can get is actually based on— it's also from the wet lab experiments. So nowadays we can have some very high throughput, like assays, for example, the phage display system, right? So, we can actually test many designed either proteins or antibodies, like, at the same time, right? So, by doing this, we can actually get the properties of the protein sequence or antibody sequence, for example, enrichment or binding affinity, right? So these are also sequence data, but for this sequence, for each protein or antibody sequence, we know their functions. We know their activities, right? So we call this data like a label sequence. Okay. So these are one type of data, which is mainly about sequence, either unlabeled or labeled. Okay. So another type of data, which is more about structure, right? So nowadays we have seen huge progress of like techniques based on Cryo-EM, especially after combination with AI like AlphaFold 2. So we are now, now we are able to solve the protein structures and more and more easy, right? So you can see the number of datasets in PDB is growing really very fast speed. Okay, so this another type of data structure data. Okay, so, so that's what we, we have, what we have in biology really, like the number of datasets in biology is really increasing at a very, very fast speed. Okay, so now we have seen the two revolutions in both fields, and then that's why I think now is really the best time to work on like generative AI for protein design. Because like for protein design, you know, the goal of like doing protein design is really like we want to design new proteins. New protein sequence or new molecules, right? And then, so we want to develop some machine learning model which is able to generate new protein sequence or new molecules. And that's what— and that's exactly the capability of, like, generative models, generative AI, right? So that's why I think here I have, like, a blueprint on generative AI for protein design in the future. So we pretty much— we can have, like, So pretty much we can have a similar model like ChatGPT for protein design. So here we can also develop pre-trained generative models. So here, what the data we have is we can have a lot of— so we have a lot of sequence, the protein sequence or antibody sequence. But besides this sequence data, what do we have? What do we have that this sequence data we have in ChatGPT, which the text and also the code. So here we have a protein and antibody sequence. But beside the sequence data, so we also have a lot of structured data, PDB data, right? So what are we going to do is we're going to train huge generative models based on the sequence and the structured data. And once we have this model, so we can start to protein design, for example, given antigen. So we can design new antibodies. Given a protein, so we can also generate protein binders. And that's the pre-trained model we have. Of course, this pre-trained model maybe will be different from what we have in ChatGPT because in ChatGPT, we just need to generate sequence. But here, because we need to model both sequence and the structure, that's why we also need to make use of geometric deep learning techniques because we need to develop generative models which is able to not only generate sequence but also structures. So that's a pre-trained model. So remember that in ChatGPT, once we have a pre-trained model, this model is further fine-tuned by interacting with humans. And by receiving the feedback from the humans, so we can fine-tune the model for the conversation purpose. So here, of course, if we want to develop ChatGPT in biology, we cannot let the algorithm interact with humans because you give humans a set of protein or antibody sequence, humans cannot give you the feedback. But instead, here, actually, we have an alternative option. Actually, it's a better option, which is so we can let algorithm interact with our wet lab experiments. So let's say here you have an AI model. So you can design some antibody or protein sequence. So we can test all those sequences in the wet lab. And the wet lab, by assays or experiments, will give us feedback which sequence has functions, what kind of properties each sequence have. And those information will be treated as feedback or training data to further fine-tune our model. So that's the high-level idea of developing similar generative models or ChatGPT. In biology, specifically for protein design. So let me give you an even more concrete. At the end, I think we can have an application like this. So instead of doing conversation with humans, we can— let's say here is an example of antibody design. So in the input, humans can input the antigen sequence or structures. And then our generative models will generate a set of antibody sequence. And then here is— we are not going to get the feedback from the humans, but instead, we're going to get feedback from the wet lab experiment. Experience. And based on this feedback, the model will be further fine-tuned and also generate or design a new round of like sequence, antibody sequence, for validation. Okay, so we can do multi-rounds of this kind of interaction or conversation, and at the end we can get the antibody sequence we want. Okay, so that's a high-level idea. So next I'm going to share some of our work we have done in generative AI for drug discovery. So I will start with some work on small molecules and later we'll move to some work on protein design and also antibody design. So the first work we did in our group, also what we have been focused on working on is predicting the 3D structures of small molecules. So another problem maybe you guys are more familiar, which is the predicting structure of proteins, right? So for example, the work done by AlphaFold 2. So in our group, we also have been working on similar problems, but we focus on small molecules instead of proteins. So for small molecule structure prediction, so the input is the molecular graph instead of like the protein amino acid sequence. And that's our input. And then the output is the 3D structure of the molecules. And then what we want to do here is we want to generate models to generate the 3D structures based on the input molecular graph. In other words, we want to model this conditional distribution. Given the input molecular graph G, what would the distribution of the output conformation be? So we have been developing different kinds of generative models for this problem. So specifically, I want to introduce the latest work we have done, which is based on diffusion models. Diffusion models is very new. And this model is actually the first diffusion model that has been applied on 3D molecules. So the essential idea of this model is we model the 3D molecule conformation generation process as a diffusion process. Maybe some of you are not familiar with diffusion process. Let me give you some kind of idea. So for diffusion process, so we have two distributions, C0. So this is actually our data distribution. In this case is the— the 3D molecular, 3D conformations of our molecules. And then we have another distribution, which is noise distribution. Basically, just, this is just a base distribution or background distributions, most of the time just a Gaussian distribution. Okay, so what diffusion process do is, okay, so first we have a forward process, which is from the data distribution to the noise distribution. So what, how do we do this? And this is very simple. Pretty much you just add some noise, you try to, this, you try to like destroy it. The structural data. So here, specifically in this example, so at every time step, you pretty much just add some noise to the coordinates of each atom. You do this like multiple steps, and at the end, you can just get a random structure. And that's how you destroy the structure of the original data, right? And then you can map it to a noise in the noise distribution. And that's the forward process. Of course, in practice, we care about actually the reverse process, which we call the generative process. So the generative process is like this. You start from a random noise structure. And then as that, what we do is you just do denoising. So basically, try to eliminate the noise in the structure. You can get a cleaner structure. You do this multiple iterations until at the end, you pretty much then can get the stable structure. So this is pretty much similar to what you have in molecular dynamics in force field, right? So at every step, you try to update the structures according to the force field until at the end you can reach a stable structure. And that's why in our model, a very, very essential component, which we call the noisy network. So basically, every— at every time step, so we have to do denoising. So we have to enumerate the noise in the input structure so we can get a refined structure, which we call like a denoising network from ct to ct-1. So essentially what the model did is like, so you have like a noise structure, you want to get a better refined structure, right? So what the model is essentially doing is, okay, we're actually trying to learn the forces over the atoms. Not the pseudo forces, right, over the atoms. And then later, and we can actually move the, the coordinates of, of the atoms according to the forces we have learned. Of course, the forces we have learned must, must satisfy a condition which is rotation and translation covariant, which means, okay, if you rotate the input structure, the forces over the atoms must be rotated accordingly. So that's the essential idea of diffusion model, which is essentially we try to train a denoising network. We try to refine the input structure until we reach a very stable structure. And here are some examples. So here you have different input molecular graph. And our model is able to generate different stable conformations. So here is actually a very concrete visualization to show you how our model is able to find the stable conformations. So in the beginning, you see that the coordinates— so that's the end. So in the beginning, you see that the coordinates of all the atoms are actually randomly initialized. And then at every diffusion step or denoising step, so we try to refine the structures of the molecules. Essentially, we try to update the coordinates of each atom until it gradually converges to the stable structures of the molecules. So that's the first work I want to introduce, which we use generative models to predict or generate the 3D structures of a small molecule. So that's one work. And then what we did later is we first actually extend this work. So instead of trying to predict the structure of a single molecule, can we try to predict the complex structure between a protein and the ligand? And that's what we did. So in this case, The input is not only like a 2D molecular graph, but we also have the 3D structure of protein. And then we try to predict their complex structure. So the high-level idea of this work is very similar to what I have introduced. And then in this model, we actually have two components. We first have an encoder. So what we do is we try to encode both the molecular graph and also protein structures and also their interactions. So with the encoder, we can somehow capture the geometric constraints between the molecule and also the proteins and this encoder. And in the decoder, so we have a diffusion model or essentially a denoising network. So what do we do here is essentially, okay, we start from a random structure, a complex structure, and then at every time step, we try to refine the complex structure or is a denoising process. Until after multiple rounds of iterations, we can converge to a stable structure. And you can see that the highlight is very similar to what we just introduced. So here are some results for protein-ligand docking. So we compare our models with both traditional physics-based methods, which, for example, like Vina, and also some latest deep learning methods. In general, you can see that now in the old days, actually, maybe 2 years ago, the deep learning method is not able to outperform the physics-based method. But now you can see our deep learning method is pretty much like it can significantly outperform the physics-based method. In general, here I can give you a sense how well the current deep learning method is able to perform. So, here basically shows that, okay, for 60% of the cases, so, the structures predicted by the deep learning model is actually— so, if we compare the distance between the predicted structures and also ground truth structure, so, 6% of the cases the distance will be smaller than 5 angstrom. That's how deep learning model is able to perform. Of course, in this case, in the testing case, all the proteins we have seen in the training data, so then we also actually test another case. What about a protein or a target? What if this is a new protein, new target, and how would the docking perform? In this case, of course, the performance is worse. The previous, this one is 60 and then is 40%. But so this basically shows, okay, for those like totally new proteins or new genes, okay, 40% of the cases, the structure we predicted is pretty much like compared to the ground truth, like, a common structure. So the distance is below like 5 angstroms. Okay. So here shows concrete examples. Okay. So this like Green one is the protein structures, and this gray one is the ground truth, like molecule position. Okay, so the pink one is structure predicted by our model. So in the beginning, it starts from a random structure, but after a few rounds of iteration, it can quickly converge to the ground truth structure. Okay, so here basically shows the the different snapshots of the iteration of the optimization process. So here I want to show that actually at every iteration, at every, every updating step, so besides predicting the current position of the molecule or the structure of the structure, we actually also output a confidence score. So basically to show you, okay, how confident we think about this prediction. So this is similar to what we have in AlphaFold 2. We have metrics like PLDDT to show the confidence. But here, we also actually did a similar metric to predict the confidence of the current prediction. So that's on predicting the complex structure between protein and also ligand. So we first— and then we first extend this kind of idea to protein design, especially on de novo protein design. You know, like for de novo protein design, the essential idea is we want to design totally new proteins with new structures and the sequence. So, there were a lot of like excitement work in the past 2 years, I think, especially by David Baker's group at the University of Washington, right? For example, like the end of last year, they actually released a new model called like IF-diffusion for protein design or for mini-protein design. So what they essentially did is actually they trained a diffusion model or joint model on the entire PDB dataset. And then based on that, they can design new proteins. So actually, their algorithm is pretty much like a two-stage algorithm. So what they did is, okay, they first generated the structures. Let's say here I have like a binding protein target. And then what the algorithm did is the first is going to generate the structure of the binder. And then once they have the structure of the binder, they're going to further use another model, which is a sequence design model. So they use ProteinMPN to further design the sequence of the binder. So in other words, the algorithm is sort of like a two-stage algorithm. And then so the structure generation model, which is called the diffusion model. So this diffusion model is very similar to the model they developed for protein structure prediction is the RoseTTAFold. So in practice, how they train this like diffusion model is to first use the pre-trained RoseTTAFold model, and then they fine-tune the model for structure prediction. So we are not going to the details, but that's the high-level idea here. And then so in our group, we also work on this problem. So we propose another algorithm for de novo protein design. So instead of trying to use a two-stage algorithm, first generating the structures and then sequence, we actually propose a new model, a new diffusion model, which can simultaneously generate the structure and the sequence. So the key idea of this model is still based on the SODWAC denoising network. So in this denoising network, we actually have three inputs. The first one is the current. Protein sequence and also the current protein structures and also some contextual information, for example, the target structure, right? And then after this denoising network, we're going to update the protein sequence and also the protein structure, maybe also update the repetition of the contextual information. So we can do multiple rounds of this denoising until it's the end. So we can get a stable protein structure and the sequence. And that's a high-level idea. So here I show you some examples of what we did based on this new algorithms. We do antibody design and also some protein design. So here we actually designed the loops, the CDR loops of antibody. Okay, so we have, let's say we have antigen and antibody complex. Okay, so what we did is we're going to mask out the original CDRs, and then we're going to generate totally new CDRs both the structures and sequence, we found that actually the new CDRs, the new C structure and sequence we have designed, so those structures we designed actually very similar to the reference complex structure we have, which means that actually are always able to recover the natural complex or the natural interactions between antigen and antibody. So that's one example. Another example, we also do something like protein design, for example, in this case. So the red one is a native protein and this is its loop. And what we do is actually we fix the other, for example, the beta sheet, we try to redesign the loops to see whether we can design longer loops. And that's one example. Another one is like we try to design novel beta-barrel proteins with different size. And this one, we actually designed the transmembrane protein complex. With a custom number of alpha helices. And then so in our group, we also actually released an open-source machine learning framework for protein understanding and design. So we really hope this kind of open-source effort or build up open-source community can accelerate the progress in this domain. So particularly in this open-source framework, We provide a very general and also flexible and modularized framework for protein design. Okay. So it provides a lot of benchmark datasets and also machine learning and models frameworks. So if you have people work on biology, so maybe you can easily leverage those machine learning models for the problems in your downstream task. And this is actually a collaboration between NVIDIA, Intel, IBM, and also my startup like BioGeometry. So finally, I want to spend a little bit on what we are doing in the startup like BioGeometry, which focuses on using AI or generative models for antibody design. So one problem we work on is called antibody optimization. So the problem is like this. So we have like an antibody, or we know the complex between an antigen and the antibody. And then we want to optimize antibody to improve the binding affinity. So essentially, we want to make some mutations, right? So we want to make some mutations in the complex. So we actually developed some similar models like ChatGPT for this task. So we also have pre-trained models. So we pre-train our models based on the protein sequence and antibody sequence, as well as some protein complex data. And that's the first pre-training model we have. And then we first fine-tune the model based on some supervised data. So, here we use a SCAMPA dataset. So, in the SCAMPA dataset, there are many protein complexes. And for each protein complex, there are some mutations. And for each mutation, we know the DDG, basically the binding affinity change. So, we fine-tune our model based on this dataset. So, then now we have a model for protein optimization or antibody optimization. And then we first select, let this model interact with the wet lab experiments. So we can design some antibodies, and so we can test those antibodies in the wet lab, and then we can send feedback to the model. So we can do these multi-runs, and then after a few runs, we are able to get optimized antibody. So here, let me show you one experiment we have done. So we actually try to optimize SARS-CoV-2 antibody, which is the CR3022 antibody. So we know the complex between this antibody with RBD complex. And then we use our models to design. So in each round, we design 10 antibodies, and we send the antibodies to the wet lab to test. And then after each round, we pick the best ones. And then we combine the— in the first round, we just do single mutations. And in the second round, we do like double mutations or triple mutations. So, after 2 rounds, we showed that actually for the Delta-like variants, we see that the binding affinity of the new mutants is able to— the binding affinity is pretty much like a threefold of the binding affinity of the wild type. And then this on the Delta variant. And on the latest Omicron variant, we showed that actually the binding affinity has improved 9-fold. And we only actually changed 3 mutations. And that's on antibody optimization. So another thing what we are doing is really like we believe that the future is really like de novo protein design and also antibody design. So in this case, so we are given input protein structures or antigen structures. We can specify. The epitope. So for example, this RBD-like antigen, so there could be like 3 different epitopes in this antigen. And for each epitope, we're going to design some antibodies. Okay, so this one is just an example. The green one is the antigen. And for different colors, the yellow one, this like Purple one. So these are different antibodies binding to different epitopes. Okay. So in practice, maybe you say, okay, maybe in some new targets, we don't know the structure, right? So in this case, we can leverage the structure prediction model, for example. So in this case, okay, so input could be a sequence. We can further predict the structure of the antigen or the target based on like models like AlphaFold 2. And then we can further do, you know, antibody design based on the predicted structure. So that's pretty much what I want to say today. Maybe I will conclude with one more minute. So basically, I think nowadays is really like, I think, the best time for people who work on biology because now we see revolution in both AI and biology. So in AI, we see huge progress on generative models. Large language models and also geometric deep learning. So which means, OK, so we are able to develop models to generate new data, new molecules, new protein molecules, right? And in biology, we have a lot of data thanks to DNA sequencing and also synthesizing. So we have a lot of protein sequence, antibody sequence, as well as structures, right? And then since we can combine the capability of generative AI and also the amount of data we have in biology, right, so we can pre-train generative models on the huge amount of protein sequence and structure data we have so that we can generate totally new protein structures and sequence. And then we can further test those new designed sequence and structures in the wet lab so we can get some feedback. We can further fine-tune our model. So by doing this, so after a few rounds of interactions, we can get the proteins we want. Okay, I think in the future, this is not only going to make impact in drug discovery, but also in other fields like agriculture and food, material, energy, and climate, considering the importance of the functions protein make in real life, right. So with that, I will stop here and maybe leave some time for questions. Thanks very much.

Thank you so much, Dr. Tong. That was such an interesting presentation. And A lot of exciting work going on. I'm wondering if you could just talk about what some of the disadvantages are of the de novo designed antibodies in terms of, would they be more likely to be eliminated by the immune system because they're not naturally occurring?

Right, right. That's a very good question. So many people ask me the same questions, like especially people work on biologics, right? So because They feel like because nowadays, you know, like many of the antibodies actually directly discovered from by doing immunization on animals, on humans, right? People believe those are more natural, especially the antibodies we got from humans, right? So, there will be no immune response. Okay. So, the question— the answer is that actually nowadays, With AI techniques, we are able to design antibodies pretty much very similar to antibodies generated from humans. It's like language models, right? So the language models are trained from a huge amount of text data. And as end, so the models are able to generate text which are very realistic. So here we have a similar idea, OK? So we can train the model on pretty much on a huge amount of, let's say, human antibody sequence. As a result, actually, the antibodies we generate from the model will be very similar to the antibodies from humans. So in that sense, actually, for the immune system, it's pretty much very difficult to distinguish, okay, which ones are from humans and which ones are designed by our algorithms.

Right. Right. Okay. That's great. So we had a question come in. Aside from increasing affinity, Can the generative model improve specificity, for example, eliminating cross-reactivity with relevant human antigens?

Yeah, that's also a great question. That's also a question I usually got people ask me. Okay. Yeah, I think it's really— so that's pretty much like, I think, as when we try to either do, I mean, antibody design or optimization, I think it's really like multi-objective, like, optimization process, right? So which means, okay, when you try to— so pretty much like when you're trying to do this design process, so you're going to consider multiple dimensions. You want to generate antibodies or proteins which satisfy multiple dimensions, right? So, and that's something— yeah. So in other words, actually in practice, we always have to design multiple predictors. For example, you have a predictor to predict the immunogenicity. The stability, affinity, or other properties you care, right? And then you can let the model pretty much generate many. So the model is pretty much like a hypothesis generator, right? You can generate many hypotheses, right? And then for each hypothesis, okay, you can use your predictors to filter. At the end, you can just keep the ones which satisfy all the criteria you specify, right?

Right, right, right, right. Okay, okay. We had another question on whether you can use this approach to improve on the immunogenicity of the antibody.

Yes, that's what I was saying. Nowadays, pretty much people train huge language models on the human antibody so that we can basically evaluate whether this antibody is from human or not. Yes. Yeah.

Okay. And we had a question on basically using this process to understand the dynamic processes that are happening with virus evolution. So if you're looking at SARS-CoV-2, for example, there's more than 10 million sequences of the virus that are available, right? And they provide multiple instances of each mutation in each context of different virus confirmations. Could the, could the methods that you've described help provide some connection there between the evolution of the virus and the sequences that we have to understand virus evolution?

Yeah, I think that's— I think some people have done that. So basically just pre-train the language. So they have some pre-trained language models on the entire protein sequence data set, and then they further fine-tune the models based on the SARS-CoV-2 mutant sequence. And pretty much They capture the— by training a language model on the SARS-CoV-2 protein sequence, I think they can somehow capture the co-evolutional information or sort of like— yeah, I think they capture the co-evolution information, so they can make the prediction. I think there are some papers around that. Actually, they show that they can predict that, make the prediction quite accurately. Yes.

Yeah, that's really interesting. And it seems to me like the SARS-CoV-2 antibodies the optimization of those against, for example, Omicron variants is a really great way to try this out, right? Since we've run out of monoclonal antibodies that can actually be used in immunocompromised people to prevent them from infection. Is that something that's going forward to be tested clinically by any chance?

Yeah, I mean, so we haven't moved forward for that antibody, but we are indeed thinking about collaboration with others on other, like, more promising candidates, because for that one, we're just like basic to test the capability of our models. But indeed, actually, so I think that the model has a good capability, especially because now we are able to model the antigen and antibody where we are. I think a very nice thing we actually want to work on is really basically trying to optimize the binding affinity of a specific antibody across multiple different variants so that you can do really like cross-neutralization across multiple antigens, right? And that's something we really want to work on.

Right, right, right. And then we had a question on using this to design anti-idiotypic antibodies.

I'm not sure quite familiar with that.

So yeah, no, that's okay. That's absolutely fine. Maybe we have time. Yeah, for one more question. We had a question on whether "Journal peer reviewers will need to have their own AI system to check the validity of models proposed for publication, or will other validation methods be used?" So they want to propose a model to validate. To validate the model.

A model for model validation. Yeah. I mean, they can do that condition on that. They have a better model. So they have more data and also better model. I mean, usually at least you need to have a better model because usually like, you know, for people work on like in the AI community, so they can develop very good models, but they may do not have like a lot of data. So maybe you need to have more data in order to be more advantageous.

I think so. I think so. All right. Well, thank you so much, Dr. Tao, for sharing this. Really fascinating and innovative research. Like you said, we really are at such an exciting time in both AI and biology. Thank you also to the attendees for participating in today's webinar and for submitting questions. We're so glad, as always, to have you join us, and we look forward to many more engaged discussions. As you probably know, we kicked off the Immunome Lab Meeting webinar series last year, so if you've missed any of the previous talks, I encourage you to visit our website, humanimmunoproject.org, to view the recordings and to find out about some upcoming sessions as well. And while you're there, please do sign up for the newly launched Human Immunome Report, which is another platform for sharing the research and voices of scientists working at the frontiers of immunology and AI. And finally, please don't forget to visit our website and follow us on social media where we will post a recording of today's today's webinar. Thank you again so much for participating today, and we look forward to seeing you soon. Thank you.

Thank you very much.