Scientist Stories: Alexander Kroll, Predicting small molecule substrates of enzymes with ML

Okay, hello everyone. Welcome to the ML for Protein Engineering seminar series. For today, we have the pleasure of welcoming Alexander Krall from Heinrich-Heine University. Alexander is a current postdoc in the research group Computational Cell Biology. At Professor Martin Lehe at the Heinrich-Heine University in Düsseldorf. And today he'll be presenting a general model for predicting substrates of enzymes based on transformer networks and gradient boosting models. Alexander, please.

Great. Yeah. So hi, everyone. Thank you very much for your introduction and your invitation to the seminar series. I'm really excited to present today. So why are we interested in predicting substrates for enzymes? More than 99% of all sequenced enzymes have no experimental evidence or high-quality annotation of their function. So we wanted to develop a tool that helps to determine the function of those enzymes with yet unknown function. As you probably know, enzymes are proteins that catalyze chemical reactions, which means they massively speed up reaction rates up to over a million times compared to spontaneous reaction rates. And every biological cell contains at least hundreds of different enzymes with different functions. And over the course of evolution, each enzyme has evolved to efficiently catalyze one or more specific chemical reaction. In this example here, we have the enzyme lactase, which can bind to its substrate, the milk sugar lactose, which is broken down into two smaller sugar molecules, glucose and galactose. So why is it even relevant that we know the substrate of an enzyme? On the one hand, enzymes play a crucial role in cellular metabolism. And hence knowing the function of enzymes is, for example, important in medical and pharmaceutical research for the study of metabolic diseases and drug development. And on the other hand, enzymes also play an important role in biotechnology and bioengineering, for example, for the synthesis of drugs, chemicals, biofuels, or in food processing, or in the degradation of waste materials. For these tasks, it's important to find enzymes or to find new enzymes suitable for processing specific substrates. As I already said at the very beginning, unfortunately for most enzymes, it is unknown which reactions they catalyze. Let's, for example, look at the UniProt database that contains more than 36 million protein sequences labeled as enzymes, but more than 99% of these enzymes lack high-quality annotations of the catalyzed reaction. So if we don't have experimental evidence for most enzymes, can we instead use prediction tools to infer their function? We can. For example, there exists the the CLEAN method that was presented 4 weeks ago in this seminar series, which can be used to predict EC numbers for enzymes. However, while EC numbers tell you which type of reaction an enzyme catalyzes, it may lack insights into the enzyme's interactions with specific substrates, which means that it doesn't necessarily provide insights about the specific molecules an enzyme interacts with. To go beyond this, a big step towards determining the specific catalyzed chemical reaction of an enzyme would be to first predict the substrate of the enzyme. However, all previously existing methods for predicting specific substrates can only be applied to small enzyme classes. This means those models were trained on many experimentally validated enzyme-substrate pairs for specific enzyme classes and cannot be applied to enzyme classes without many experimentally validated training data points. In contrast to these approaches, we wanted to develop a general prediction model that can predict substrates for any enzyme, even if not many training data points are available for highly similar proteins. More precisely, We wanted to develop a binary classification model that can receive as its input any enzyme and a small molecule and predict whether the small molecule is a substrate for the enzyme or not. We use the GO annotation database for UniProt IDs to extract 18,000 experimentally validated enzyme-substrate pairs. Consisting of more than 12,000 unique enzymes and 1,400 unique substrates. Since we want to train a binary classification model that either predicts the positive or the negative class, we also need negative data points. That means enzyme-molecule pairs where it is known that the molecule is not a substrate for the given enzyme. Unfortunately, such data is not stored in large protein databases. And thus we had to find a different way to get negative data points. On the next slide, I will talk about how we achieved this. As I said, in the literature we can mostly only find positive enzyme-substrate pairs. So how can we get the negative training data? We have 1,400 different substrates in our dataset. And let's imagine all of these substrates are displayed on this slide, where more similar substrates are closer to each other. And for every given enzyme in our dataset, we have one or multiple positive substrates here displayed in green. I hope you can, you can see it well. And although many enzymes can be promiscuous, which means They can have secondary substrates for which they did not evolve, but to which they can still bind and can catalyze. Among this limited set of molecules in our dataset, enzymes are quite specific towards their substrate. That means if we randomly pick molecules from this set of substrates, there's a very high likelihood that these molecules are not substrates for the given enzyme. So what we did is we randomly selected 3 negative molecules, here displayed in red, for every positive enzyme-substrate pair. We decided to generate more negative data than we had positive data because this not only provided us with more training data, but it also more closely reflects the true distribution of positive compared to negative data points. However, when we use this data, we found that the model, or the trained model, is not very good at distinguishing between substrates and molecules that are structurally very similar to the true substrate. To overcome this issue, we came up with the idea of considering only those molecules that are structurally similar to the true substrate. And we sampled only from this subset. This resulted in a more difficult learning task that forces the machine learning model to learn the important features for distinguishing the true substrates from structurally similar non-substrates. This sampling process resulted in a dataset with a total of 70,000 data points which we divided into 70% training data, 10% validation data, and 20% test data. And while splitting the dataset, we made sure that the enzymes in the training and in the test set are not too similar. So now that we have our data for training and validating the model, let's have a look at the general prediction pipeline that we wanted to follow. We want to build a binary classification model that predicts if a small molecule is a substrate for a given enzyme or not. So to train a machine learning model, we first need to map the enzyme and the potential substrate to numerical representation. Then we can concatenate both vectors, which results in a vector with both information about the protein and the small molecule. This vector can be used as the input for machine learning model that can be trained to predict if the small molecule is a substrate for the enzyme or not. On the next slides, I will tell you how we encoded information about the potential substrate and about the enzyme in numerical vectors. Substrates of enzymes are typically small molecules such as the one on this slide. You can probably see that such a molecule can be interpreted as a graph by interpreting the atoms of the molecule as the nodes of a graph and the bonds of the molecule as the edges of the graph. To add information about the molecule to this graph, feature vectors are then computed for atoms and bonds, which contain information like the atom type, the atom mass and charge, or the bond type and whether the bond is part of a ring or not. There now exist diverse methods to generate molecule representations from such a graph structure. One set of methods of so-called expert-crafted fingerprints, where the function, where the functions for mapping the graph structure to single numerical vector were defined by experts. These approaches usually result in binary vectors where each entry can, for example, indicate if a certain substructure or property is present in the molecule. Alternatively, graph neural networks which are deep neural networks for graphs, can be used to process the molecule information. Graph neural networks iteratively update all atom and bond feature vectors for predefined number of steps using information about the neighboring bonds and atoms. So for example, if we want to update vector x6, we would use the information about the neighboring bonds and about the atoms X1 and X5. This results in an improved atom representation in the context of the whole molecule. After iteratively updating all vectors for predefined number of steps in this way, we can calculate the element-wise mean of all atom feature vectors, in this case vectors X1 to X12, to obtain a single graph representation, here displayed in red. In order to actually generate meaningful representation, we train such a graph neural network for a specific task. We use the red vectors together with enzyme information, here displayed in blue, as the input for a fully connected neural network, which we train to predict if the small molecule is a substrate for the enzyme or not. So this is exactly the task that we are interested in. And we trained this whole model end-to-end, which means we adjusted the parameters of the graph neural network and of the fully connected neural network simultaneously. This way, we forced the model to store all meaningful information about the small molecule that is relevant for predicting substrates for enzymes in this red vector. So after model training, we extracted for all potential substrates in our dataset the red vector and used it as a task-specific small molecule representation. To generate numerical representations of the enzymes, we wanted to use the enzyme amino acid sequence. As most machine learning models require for all data points input vectors of fixed length, we want to obtain numerical vectors of a fixed size even for enzyme sequences of different lengths. We can achieve this using transformer networks. So we can use the amino acid sequence of the enzyme as the input of a transformer network by representing the positions and the type of every amino acid in numerical vectors, and then use these vectors as the model input. In order to obtain more meaningful representations, the transformer network updates each amino acid representation by using information about the other amino acids in the input sequence. However, to actually achieve meaningful representations, the transformer network has to be trained for specific tasks. Typically, roughly 15% of these amino acids in the input sequence are randomly masked. For example, we wouldn't let the model know what type of amino acid we have at the second position. And the model is then trained to predict the type of this amino acid. This requires that the model uses information about the neighboring amino acids, and maybe also about amino acids that are far apart in the linear sequence, but which might be close in the protein 3D structure. Such a transformer network, the ESM1B model, was trained by the Facebook AI Research team on nearly 30 million protein sequences. And to obtain a single vector, we can calculate the element-wise mean of all updated amino acid representations. We can use this vector, the ESM1D vector, as a representation for the enzymes in our dataset. And this is the way how numerical protein representations are typically generated. However, by simply calculating the element-wise mean of many different vectors, we might lose important information. And on the other hand, we cannot be sure that these ESM1B vectors contain all information relevant for predicting enzyme-substrate pairs because the model wasn't trained for this specific task. Similar as for the small molecule representations that we have seen on the previous slides, we wanted to generate task-specific enzyme representations. To achieve this, we added a representation for the whole enzyme, basically a classification token, to the ESM1B transformer network., and we updated this enzyme representation in the same way as the amino acid representation. This means in order to generate a more meaningful enzyme representation, the model can use information about all amino acid representations. After the update step, we combine this updated enzyme representation with a representation for small molecules and use both representations as the input of a fully connected neural network. This network is trained to predict whether the added small molecule is a substrate for the enzyme or not. By training the full model end-to-end, which means the parameters of the transformer network and of the fully connected neural network, are adjusted simultaneously, we force the model to store all meaningful information about the enzyme amino acid sequence in this enzyme representation. When training the model, we did not train the whole transformer network from scratch, but instead we initialized its weights with the parameters of the already trained ESM1B transformer network, and after model training, we extracted for all enzymes in our dataset this enzyme representation and used it as a task-specific enzyme representation. Let's do a short recap of the steps we have done so far for building an enzyme substrate prediction model. First, we searched the GO annotation database for experimentally validated enzyme-substrate pairs. And since we don't have any negative data, which we need for model training, we sampled negative data in a clever way. Next, we created task-specific numerical representations for enzymes and small molecules, which we both want to use as the input for prediction models. We trained a transformer network in order to generate task-specific enzyme representations,, and we trained a graph neural network to generate task-specific small molecule representations. We concatenated both vectors. So now we have a vector that contains both enzyme information and small molecule information. We use this vector as the input for a gradient boosting decision tree model, XGBoost, to predict for each input if the small molecule is a substrate for the given enzyme. Or not. This results in a prediction score between 0 and 1 for every data point, and we classified all data points above a threshold of 0.5 as positive. Overall, our model achieved an accuracy of over 91% on our imbalanced test dataset. To further analyze To evaluate model performance, we divided the test sets into different subsets according to the maximal sequence identity compared to the training enzyme. For example, let us consider the subset on the left, 0 to 40% means that for all test data points in this category, the enzymes have a maximal sequence identity of under 40% compared to any training enzyme. And to put these categories a little bit better into perspective, it is likely that two enzymes share the same function if their pairwise sequence identity is above 60%. On the left side, we see on the y-axis the accuracy, but since we have a dataset with imbalanced class labels, the accuracy might be a bit misleading. Since we have a negative to positive ratio of 3:1, always predicting the negative class would already lead to an accuracy of 75%. Hence, we also computed a performance metric that is robust against imbalanced datasets, the Matthews Correlation Coefficient MCC, which is comparable to the Pearson correlation coefficient but for binary data. A value of 0 would result from a model that randomly assigns class labels or always predicts the same class, and a value of 1 would result from a model with perfect prediction. And we can see that the model still provides good predictions even if the enzymes are not highly similar to the training enzymes. We wanted to answer a similar question for the substrates. We wondered how many substrates need to be present in the training set such that we can provide accurate predictions for this substrate. To answer the question, we calculated for every test data point how often did the substrate occur in the training set as a positive substrate, and we divided the test set accordingly. On the y-axis, we have again the Matthews correlation coefficient MCC, and we can see that if a substrate is not present in the training set, This model cannot provide meaningful predictions, which is the biggest limitation of this model. However, for those 1,400 substrates present in the training set, the model can provide accurate predictions, and expanding the training set with only 2 positive enzyme-substrate pairs for specific substrates already leads to high model performance. As I already said, this is the first general model for predicting enzyme-substrate pairs, and thus we could not benchmark its performance to models with a comparable scope. However, we made a comparison to models aiming to predict substrates for specific enzyme classes. We predicted substrates for different glycosyltransferases from two plant organisms, and predictions with our model led to significant improvements in comparison to those models that were specifically designed for this task. To show you that it is worth to make the effort to train task-specific substrates and task-specific enzyme representations for this prediction task, here are the results compared to non-task-specific representations. We compared our task-specific graph neural network generates substrate representations to expert-crafted fingerprints, eCSPs, which are also known as Morgan fingerprints. And so we have on the one hand here the eCSPs, which we compare to our task-specific representations, which we here call GNN representations. And we also compared the standard eSM1B representations to our task-specific ESM1B representations, which are denoted here as ESM1B_ts. And you can see that especially for using task-specific enzyme representations, the improvements in model performance are quite high. So let's get shortly back to our prediction pipeline and let's think about how we can improve this. Right now, we generate a substrate representation without knowing for which enzyme we want to make a prediction, and vice versa, we generate an enzyme representation without knowing for which small molecule we want to make a prediction. This means we have no information exchange until the numerical representations are already generated and we concatenate both vectors. However, we hypothesize that we might be able to generate more meaningful representations if we allow the model to look at the other molecule for which we want to make a prediction during the whole process of generating this representation. To achieve this, we developed a transformer network which we call ProSmith that can process the protein or enzyme amino acid sequence and a SMILES string of the small molecule which encodes the small molecule structure in the same input sequence. We tokenize the amino acid sequence as well as the tokens of the SMILES string and map both into the same embedding space. This allowed us to use the enzyme sequence and the small molecule SMILES string as the input for the same transformer network. The attention mechanism of the transformer network now allows us to look at the small molecule when extracting enzyme information and vice versa. It allows us to look at the enzyme amino acid sequence when extracting small molecules. Information. We trained this transformer network end-to-end for the binary classification task of predicting if the small molecule is a substrate for the enzyme or not. This is already the prediction task we are interested in, and in principle, this is all we need. However, we found that if we extract the classification token after model training, and use it as the input for a gradient boosting decision tree model, the prediction performance was even a little bit higher. This multimodal transformer network can be in principle applied to any protein-small molecule interaction prediction task, and we successfully applied it also to tasks such as predicting drug target interactions or predicting the Michaelis constant of enzyme-substrate pairs. This PROSPECT model is currently under revision, but the preprint is available on bioRxiv. So let's have a look at how this new model architecture improves model performance for the enzyme-substrate prediction task. Overall, the accuracy increased from 91% to over 94% on the dependent test set. And we can also see that the model performance increases for substrates that have not been present in the training set or that have only been present once in the training set. And we can see something similar for the enzymes. We get the largest improvements for those enzymes that are not highly similar to training proteins. So overall, using this new model architecture, we were able to improve the generalization of this model. This prediction model is available on our web server deepmolecules.org, which I quickly want to show you. I hope you can still see it in the shared screen. So basically on this web server we have not only the enzyme substrate prediction model, but we also have prediction models for predicting the turnover number kcat for enzyme reaction pairs, predicting the Michaelis constant Km of enzyme substrate pairs, and also the option to predict substrates for transport proteins. And if you want to make a prediction, You always have the choice between uploading a single enzyme-metabolite pair or multiple enzyme-metabolite pairs. So let's say we want to make a prediction for a single enzyme-metabolite pair, then you can simply enter an amino acid sequence. And then let's say we enter a small molecule information. For example, by specifying the small molecule through its INCI string, then you can submit this. And then normally, if everything works fine, and for a single data point, this should only take up to 1 or 2 minutes if the web server is not too busy. And then you get a prediction score, which is between 0 and 1, where scores closer to 1 indicate, then it is more likely that the small molecule that you entered is a substrate for the enzyme. And alternatively, if you have many data points that you want to check, you can also have the option to upload multiple data points where you can upload a CSV file containing of two columns, one column for the enzyme amino acid sequence and one for for the potential substrate, and then you can upload up to 500 data points at once and also get your predictions back as a CSV file. Okay, and for example, now our job is calculated, and since I only selected the random small molecule for this enzyme, we get a prediction score of 0. Yeah, probably it's very unlikely that this is a plausible substrate for this enzyme. And you can see that we also get additional information, how often this metabolite, the small molecule that you want to test, was in the training set. Because as we have seen, the model becomes more accurate or more reliable the more often the small molecule was present in the training set. So in this case, we would have a small molecule that wasn't as often or wasn't in the training set. But if you have here a number higher than 0, this would indicate that the prediction is likely more accurate. Okay, so let's get back to the presentation. Okay.

Perfect.

So this is the web server, but also the Python code for all prediction models are available on my GitHub page. Oh, I don't know why it switched back. Okay, so I hope you can see the presentation again. Before I finish the presentation on this last slide, I want to shortly talk about how the model could potentially be further improved. I assume that one option could be to further explore how we should sample the negative data points. I told you that we explored different strategies by sampling hard negative data points, meaning we sampled molecules structurally similar to the true substrate. We also tried to simply randomly sample from the whole set of substrates in a our dataset. One could further explore sampling strategies that might be more suitable. For example, we could explore if some substrates should be excluded from sampling for specific enzymes. For example, if we have a very similar enzyme with a specific substrate, maybe it would be beneficial to include the substrate for all highly similar enzymes as a negative data point to avoid sampling false negatives. And one could also explore if 3 is already the ideal number of sampling negative data points for every positive and substrate pair, or maybe the number of negative data points should be higher, or this number could also be maybe randomly distributed. Another potential way to improve model performance could be further improving the numerical representation. For example, for the enzymes, we only use features extracted from the amino acid sequence, but with AlphaFold 2, we can predict the enzyme 3D structure for almost all the enzymes we are interested in with a high accuracy. So maybe it could be beneficial to include features extracted from the 3D structures as model input. And lastly, one could further work on the model architecture. We've seen that our new model architecture led to a large improvement in model performance, And maybe there's even more room for improvement by further adjusting the model architecture or by combining multiple different model architectures in an ensemble model. And with this, I'm at the end of my presentation. And on this final slide, you can see the people who worked together with me on this project.

Thank you very much for listening. All right, thank you very much, Alex, for the talk. If anyone has any questions, feel free to leave them in the chat, or you can also feel free to unmute yourself and ask directly. There's one question in the chat right now from Christine asking, what is the size limit for the metabolites?

Currently, the size limit for the metabolites is, since we use a graph neural network, we had to set a size limit during training for the number of nodes, meaning for the number of atoms, which is 100. We had very few which were over this range of having 100 atoms. But after model training, we can still apply this graph neural network even to larger substrates, but it wasn't trained on these substrates, so maybe this model is here and not so reliable. But yeah, in principle, there is no size limit for the metabolite as long as it is feasible to still process it with the graph neural network. And of course, since we didn't train on so large metabolites, yeah, probably if you choose too large metabolites, prediction accuracy will become lower.

Great, thanks. A question from Jeff. Do you have many multifunctional or promiscuous enzymes in your training set? Have you evaluated your model specifically on enzymes that can act on multiple substrates?

Substrates? I think we don't have so many. I mean, I don't know if they're multifunctional or promiscuous. I mean, I guess most enzymes are also promiscuous, but what we used here is that among the molecules that are most commonly present in biological cells, enzymes are quite specific towards their substrates, at least what I know. But I think outside, if you take the whole molecule space, I think enzymes are quite promiscuous. So I think we definitely have promiscuous enzymes, but I guess this is not what you're asking. So at least we don't have many enzymes with multiple functions in with multiple different substrates in our dataset. We extracted 18,000 data points from the GO annotation database, and we have over 12,000 unique enzymes, so which means on average we have 1.2 substrates per enzyme. And if we consider it that we also probably have a lot of enzymes with reactions that have multiple substrates. I assume that we don't have so many enzymes with multiple reactions in the dataset, although I assume, yeah, many might be multifunctional. And yeah, we kind of evaluated the model on enzymes that can act on multiple substrates. When I showed the results for the comparison to the model that specifically was trained to predict glycosyltransferases for two plant organisms, there we basically had, I think, 50 different enzymes, which were all scanned a range of— actually, I forgot the exact number, but again, the range of different substrates. And there we also had enzymes which had multiple substrates. But yeah, you can see also that the accuracy is okay, an MCC of 0.5 isn't too bad, but maybe the reason for why the accuracy isn't higher, that maybe our model was only trained to predict the primary natural substrate of the enzyme. And then maybe it has more trouble to identify secondary substrates which it maybe can bind, but to maybe which it did not evolve to catalyze. I hope this kind of answers the question.

Great, thanks. And also feel free to unmute and follow up if you have any follow-up questions. The next question is from Suvi. Do you think balancing the data could have improved the model performance?

I'm not sure to which step you refer. So if you refer to when generating the negative or when, yeah, in the overall dataset, it's basically our choice how much negative data points we want to sample for every positive and lambd-subscript pairs. And we also played around with different numbers of negative data points. But yeah, I guess this is a little bit tricky because at the beginning we only sampled one negative for every positive, but this didn't result really in good prediction performance also similar to what I said, that the models are not very good at distinguishing between the two substrates and between substrates that are structurally similar. And I guess because if you only have one negative for every enzyme, then, and one positive, then I guess it's really hard for the model to learn, okay, what are actually the features of the positive substrate that makes it a positive substrate for this enzyme? Compared to what are really the features of the negative molecule which don't make it a substrate. So I think in this case, it definitely helps to have more negatives. We also played around with the number of negative data points, and we found that 3 is not too imbalanced to train the model. And if your question is not regarding the generation of the dataset, but the model training, we also played around with this. So when training the task-specific enzyme representations, we also tried to sample batches during training such that we have a balanced distribution, but this didn't result in better model performance. So we trained also the task-specific representations on the imbalanced dataset. And also for the gradient boosting model that we trained in the end, we— there it was actually a problem for model performance in the end if we had such an imbalanced dataset, but we found that it helped that we set a weight to the negative data points. So while generating new trees in the gradient boosting models, there's a lower weight to the negative data points, but we still have the advantage of seeing different multiple negative data points for every positive substrate. But yeah, I assume there's still many different ways one could explore how to better handle the negatives or different numbers of negatives or to train the model better on on an imbalanced dataset.

Okay, great, thanks. The next question is from Alan. It's a bit long, so maybe you can read the comment as well. So thinking of possible improvements, what about the training data? Did you try using substrates described by reannotations in UniProt? The reason being that not all GO terms have mappings to chemical structure data, and other data sources might have more. For instance, there are over 8,000 different reactions in REA that are linked to enzymes in UniProt, but that have no corresponding GO terms. It's also worth noting that most GO molecular function ontology terms for enzymes are largely based on EC numbers, so it won't give you high level of specificity.

Okay, so we— okay, I will try to start at the top. So I guess the training data can definitely be improved and increased since we only use the Go annotation. Database for UniProt IDs, and since you pointed out there are also other sources for gathering experimentally validated enzyme software pairs, so yeah, the training data is so crucial. I guess this is definitely one way to improve the model. And actually, we tried to find the correct substrate from the GO entries using whatever information we had. If we had a rare reaction ID, we extracted— tried to extract the substrate. Or if we don't— only had the reaction equation, we, we tried to extract the substrate names and try to map these using PubChem to the chemical structures. So yeah, if we couldn't map, couldn't actually find the exact chemical structure of the substrate of this enzyme, we didn't use this data point. And we didn't use the REA database And I didn't know it was linked to UniProt IDs. Yeah, sounds definitely interesting as another source for more training data. And I guess it's important that those are actually experimentally validated to not add too much noise to the training data. So I don't know if you know all of these data points, uh, have experimental evidence. Um, and okay, so I think that the GO terms only refer to the EC number. I think we only, we only use data points that actually refer to the exact chemical reaction that was catalyzed. And I think since we only use data points with experimental evidence, I think in these cases this should definitely always have been the exact reaction that the enzyme catalyzed, but yeah, if you think this is not the case, then maybe this is worth discussing it. So yeah, maybe this is an area also to further improve the model.

Cool, thanks. There are two questions from Rana. The first one is, does your model consider enzyme complexes at all? And the second is, have you done any analyses that look at the attention scores from ProSmith that might give you insight on what the model is learning?

Unfortunately, we don't consider enzyme complexes. So from the GO annotation database, we get one unique protein ID and we took the amino acid sequence that was referring to this UniProt ID. And the second question was, no, but I definitely, so the second question was if I looked at the attention scores from ProSmith to see what the model is learning. I haven't done this in detail yet, but I definitely want to do this. And this is a question I already got quite often. So this is also what we want to put on the web server in the future. So basically, yeah, if I understand it correctly, is if we have our multimodal transformer network, then we get the attention scores for the enzyme amino acid sequence. So we see which parts of the enzyme are most relevant for making a prediction and most likely This refers to the active site. So yeah, I guess this is valuable information, and in the future we also want to use this information, or at least give it also as an additional output on our web server.

Cool, thanks. The next question from Stepan: Is the model only suitable for predicting the interaction with substrates? Or with cofactors as well? For example, can it tell the difference between ATP or GTP-dependent enzymes?

We also trained the model on cofactors. So when generating the dataset, we took whatever we found on the left side of the reaction equation as substrates. So this definitely includes ATP and GTP. So ideally, our model also learns to distinguish between ATP and GTP-dependent enzymes. And yeah, one of the most common substrates actually cofactors that in our dataset was ATP, and actually distinguishing between enzymes that use or use not ATP as a cofactor had actually quite high model performance. So I assume that our model can do this quite well.

Cool, thanks. And the next question is from Darren. Did this model predict the enzyme's active site in order to classify substrates?

No, so basically what we only used information about the enzyme amino acid sequence as input for, well, yeah, to extract information from the enzyme and basically the attention mechanism of the transformer network allows the model to only focus on those parts of the enzyme sequence which are relevant for making a prediction for whether the small molecule that we added is a substrate for the enzyme or not. So because the model has the capability to expect on the relevant parts, I assume the model focuses mostly on the active site of the enzyme, but we didn't directly encode this into the model. So this is hopefully something that the transformer network learns while we train it for this task.

Cool, thanks. There are a few suggestions from Alan on how to potentially vary or expand and the training data. I think, I guess, yeah, you can sort of read through that as well. But I think the next question is from Shen, who asks, can active sites or catalytic residues, although sparsely available, be utilized to improve model accuracy or interpretability?

I'm not sure. I recently have seen, um, that there are models for also predicting the active site, um, of proteins or enzymes. So I think using only like experimentally validated enzyme active sites probably does not give us enough data, and then we, we only have a very small dataset and probably it's very hard to learn from this. But I'm not at all familiar with those models that can predict the binding site or the active site of enzymes. But if they should be accurate, then maybe it's a good idea to give this the model as an additional information. Yeah, I'm not quite sure how one would do this, but as a sequence, yeah, maybe one could find the net domain that only contains the active site. And omit the rest, so the model maybe has less noise for the predictions and could see if this further improves this or somehow indicate the model in another way what are the residues belonging to the active site, yeah, or maybe even use AlphaFold 2 to predict the 3D structure of the enzyme and then only use a representation maybe of the active site of the protein, but yeah, I'm not quite sure how to do this. But if the active sites or active site information is available for many proteins, then I guess it could be valuable information to somehow add this information to the model, although I don't really know how.

Cool, thanks. And one more question from Jiting: In the presentation here, the classifier itself is a multilayer perceptron, but in the paper it seems to be a decision tree or random forest. Was there any difference in performance when using these two models?

So basically for this multimodal transformer network, this was trained first in this way and then using this classification token as the input for a multilayer neural network to make a prediction. And this already gave quite good results. But then we found if we extract this representation and use it as the input for a gradient boosting model, which we train newly on this representation, then the model performance was even a little bit higher. And I assume there are two potential, I don't have the exact numbers in mind, but the difference was between using this end-to-end trained transformer network with the multi-layer perceptron in the end on the gradient boosting model. But the difference was not super high, but it was significant. And I feel, or I have two potential reasons in mind for why this might be the case. On the one hand, by training this whole model end-to-end, training this transformer network and then the fully connected neural network simultaneously, it might be, it has to focus on two different tasks, right? It has to extract meaningful representations with information about the enzyme and the substrate and directly adjust the parameters to adjust the parameters of the fully connected neural network. So maybe it's a little bit more difficult to focus on both tasks simultaneously. Oh, there are balloons in my video. Compared to splitting this task, generating this representation, and then using this as the input for a new model that only focuses on the generated representation. But I'm not sure about this. Maybe some people experience training transformer networks end-to-end. Can comment on this, or this is maybe a problem during model training. And if this is not the reason, then maybe it is just the case that a gradient boosting model is more suitable than a fully connected neural network to extract the most information from this numerical representation that we created or generated using this multimodal transformer network. And since you were referring to the paper, I don't know if you refer to the enzyme substrate prediction model, but basically here we did the same, for example, for the substrate, we first trained this model, extracted this representation, used it as the input for graph neural network, and also using this transformer network, we did the same, this model end-to-end, but then extracting this representation and combining this with our task-specific captured representation as the input for a gradient boosting model that better resolves.

So I hope this answers your question. Okay, great. Thank you, Alexandra, for that great talk. In the interest of time, I guess we'll wrap up the talk here. So thank you very much for all your questions and attendance. And we have our next talk scheduled in 2 weeks. The schedule will be up on our website pretty soon. On the PTM member model. So I hope to see you, uh, everyone in the future as well.

All right, thanks everyone. Thank you.

Bye.