Scientist Stories: Fabian Theis, Machine Learning and Systems Biology

So, okay, so good afternoon and thank you very much for joining us at the seminar series, distinguished seminar series of the Cambridge Centre for Eye and Medicine. I'm Mihaela van der Schaaren, I'm the director of the centre, and it is my great pleasure to introduce today Professor Fabian Theiss as distinguished seminar speaker. And he's a director of Helmholtz Munich Computational Health Center and the scientific director of the Helmholtz Artificial Intelligence Cooperation Unit. He's a full professor at TU Munich and is holding the chair of mathematical modeling of biological systems, as well as an associate faculty at the Wellcome Sanger Institute. It is really a pity that we cannot Fabian in person today. But he has been an inspiration for all of us who are interested in doing research in the area of machine learning and AI for genomics. He has been a true pioneer in this area and a thought leader. So I really hope, Fabian, that we can learn a lot from you today. And thank you very much for visiting us virtually, and we look forward to welcoming you in the future in person.

Thank you so much, Mihaila, for this kind introduction and also for the invitation. Maybe you can stop sharing the screen, I mean, then I can share mine. Yeah, I'm really quite excited to talk to you and I'm really, I mean, I think I said that before, sorry that we can't visit each other and it would have been just so much fun to actually go to one of your beautiful, cool English pubs afterwards. You can't imagine, but I'm missing pub sandwiches inside and all these things. So this is something I hope we can catch up, particularly in the beautiful city of Cambridge. Also, I wanted to sort of say today I will be talking about some of our machine learning applications in sort of state learning and single-cell genomics. I won't be connecting as much to the medical side of things, even though we have a bunch of applications, but I will sort of stay a little bit more on the method and conceptual side. And I do think there's a whole bunch of links to the study of variation across humans that in particular, I think, Michaela, you're doing with some of those time series information and so on. And we've been doing a few of these, but I won't be putting this as strong as a focus today. I hope we have opportunity to discuss this afterwards. So thanks so much for the invitation. Let me get started. And if there are questions, I'm happy to take them in between, otherwise, of course, at the end. So this is our center. I just was thinking one slide I should just briefly introduce that we've been setting this up. We have a bunch of PIs working in different directions at this Helmholtz Computational Health Center here in Munich. So this is the set of PIs, very junior set of faculty working both in health AI, the genomics imaging side, biomedical systems, biomedicine side, more mechanistic side. Similar indication, if someone wants to come over to Munich at some point again, please do so. So the question that has been driving me I think for a long time has been, yeah, how can we deal with, how can we understand how cells make decisions? I essentially started looking into this whole field from time and microscopes, very different type of data than going to the genomic side, but always been sort of following, yeah, isn't it fascinating that those little computers essentially take up external signals and then actually do decisions such as whether to survive, whether to grow, divide, or differentiate, or maybe just do nothing, or maybe just go into apoptosis. Like all this control, obviously heterogeneous fashion with different inputs. But then there needs to be some way to interpolate the set of functions to make these predictions, right? Wouldn't it be ideal, you know, in systems biology 10 years ago we had this vision of writing these virtual digital twin cells that can do everything. I think this is cool, but I think with enough data, like with a large-scale view of cellular state, maybe we can start asking this in particular under perturbations again, I think we're getting into a range where these type of data and questions could be possible. I think we could potentially approach one of these holy grails in computational biology, how do gene regulation networks and things like that work. So the tool to look into that, I mean, I said already the title is not bulk genomics, so not this smoothie, but single cell genomics side, which in particular also Sarah Teichmann, John Marioni, and many other people in Cambridge have been the pioneers for a lot of these techniques. I'm very happy stand on those to then essentially sort of deconvolve this heterogeneity that you would have in taking an average of tens or 100,000 of cells to quantify, okay, very often messenger RNA state or other type of state of a cell, but instead do it on the single cell level. One of the tricks why this works, and I hope you can see my mouse, is that we can capture single cells in these water droplets, very minuscule droplets embedded in oil, But if you capture cells there, then you do sort of a tag within this droplet, do some type of library prep, and then just throw them together. But then you have been barcoding cells, and then you can do like bulk sequencing, and you get the single-cell resolution of this. Because sequencing machines became cheaper faster than Moore's law, you know, you can do this in a large-scale fashion. So this has been, even the non-droplet version, very early on recognized as a big thing, and Nature's been named the method of the year some time ago. That has been followed up with multi-omic version, as a method of the year just 2 years ago. And then just last year, the spatial resolved transcriptomics as another thing. So you like doing this not only for dissociated cells, but for like the, the whole picture. So single cell genomics is becoming big data, and you need to somehow look now maybe with other tools than you were sort of doing in this case of this in bulk, you know, was typically what in stats I guess you would call small n, large p. So very few actual observations versus a high number of different observations in the transcript. So usually like 20,000 transcripts, something like that, versus a few and here a high number of it. So you're easily hundreds of thousands. So the question that you would ask to these things is, you know, you have cells maybe going through decisions. So this is a sketch that we had at this lifetime proposal or sort of EU consortium that we've been setting up, particularly because I keep pushing together with Jean-Yves Amouzzi to think how we— how cells make these, these sort of decisions over life. And in this sketch, you know, they sort of maybe grow old in a healthy fashion, but then they take off in all these directions. And, you know, if we just sample enough, maybe we can really learn this space, and then maybe we can really learn into trajectories where cells, just because we have it by similarity, you could sort of arrange them are maybe in the healthy situation and disease situation something goes wrong. And, you know, if we can learn that, you know, have this function of the computer just before, maybe we can even much less like go back. So that's some of my motivation. I think asking these questions and applying them to this type of data obviously needs machine learning tools for that. And so we've been playing around and doing a lot of these type of things over the years. I won't give you now sort of a big overview of things that we do, but I want to show you two sort of bigger vignettes. One will be on the smaller scale dynamics side of things, trying to understand sort of these trajectories. And then the second part will be how we do sort of neural latent space learning, which is sort of a generalization of one-dimensional trajectory if you want, by using neural network techniques for that. And if you just excuse me for a second, maybe I should close the door because the kids just came home. Just a minute. Sorry for that. So thanks so much. So, uh, how can we put this together? Before sort of answering the machine learning part, I just want to say, um, that we've been actually investing quite a bit in the infrastructure. And one of these things that has become somewhat popular is our single-cell analysis in Python tool ScanPy. This is a lot of things have been attached to. So I think that's sort of a software framework that you can play around with now, even if you're not from the field. And maybe if you're interested in sort of going from a machine learning side of view, you have also a bunch of challenge datasets, for example, NURPS presenting a problem in this direction. I think it's an interesting opportunity now. So because of these datasets becoming available and sort of scaled up, no wonder more people have been thinking about that. And here together with, well, actually, my postdoc Luke, actually in his PG time, has been sort of putting up this database, Single RNA Tools, that has been sort of listening in and trying to aggregate tools in that field that have been around. And, you know, he realized that there's been so many, many things around that at the time was like 250 methods when he was writing his first paper. Now we have more than 1,000 people doing different things. So it gets really crowded and popular. But if you look around what type of big trends are that early on a lot of people were doing visualization, ordering of cells, the clustering, and later on changes were such as, you know, you want to do more perturbations, classification, do some type of integration. So there was a shift of trends. I'll be saying a little about integration, a little about perturbation later. So my example that I want to speak about is trajectory inference. So you want to identify a cell's position on a biological trajectory using single-cell RNA-seq snapshots. So to do so, we would be looking by putting cells in order and then let trajectories run across these systems. And one way to do so is just have essentially random walks go around that, not do just one, but do multiple of these. And if you have now this ensemble of random walks, And you know, this is obviously not happening in this two-dimensional, like three-dimensional visual state, but typically in like 20,000 dimensions where you maybe sort of learn this latent space. But you know, essentially if you do this, then you can sort of give an order to these things. So if you have that, you can say maybe this is like early, this is late, you learn, so it starts going through points. So this has been called pseudo-temporal ordering, was initially pioneered by Cole Trebnell with his monocular tool. Then I paired with Wanderlust, we did something called Diffusion Pseudo-Time. And many follow-ups ever since. So this works nicely, but you know, you could do more. And I guess situations where things like this work well is, for example, for normal development. In this case, we look at a dataset called from a system called pancreatic endocrinogenesis. This would be sort of stem cells that would then go in a high state, sort of differentiated state of this ADN3 factor, and then would sort of go into alpha, beta, delta cells. This is in the pancreas, how those hormone-emitting cells are being made. This is from a collaboration where we worked together with Heiko Linkertz's lab. In this case, you know, if you specify the root cells, you can make these random walks, and then you can sort of get this distance map on top of that. This is quite different though if you're in a perturbed situation. You know, I would mention this is sort of what's happening now in the field. For example, if you're not sure where the initial state is, depending on the initial state, obviously this type of distances and trajectories would look very different. Also, if you're not sure about the directions, what's happening sort of at these places. So in order to really be good at making those trajectories, it'd be a good idea to actually sort of add more information to that. And this is, I think, sort of going this direction of this holy grail that I was trying to say in the beginning: how can we put a cell into its context and then sort of understand what type of responses it sort of satisfies? In this case, I guess you can call it learning maybe a gene regulatory network in a sense that if you say you want to understand somehow the dynamics of your particular set of transcripts as a function of the current state of transcripts. It's not sort of the general problem. It's like super hard and usually there's all kinds of approaches for that, but usually it never works. But if that worked, this f you could maybe call a regulatory network because it would be connecting one particular transcript state to some other factor. And the nice thing is now that you can actually measure in parts this left-hand side. This is something called RNA velocity. This is, I guess, one of the seminal papers in the field over the past 5 years from Linus Neukirchenko's lab, where they've been noticing that there's actually spliced as well as unspliced reads available in RNA-seq, normal single-cell RNA-seq data. And if you now plot spliced versus unspliced, you're getting sort of some cells where you have more unspliced than sort of the average, you have less unspliced. In this situation, if you have more unspliced, this RNA is just being made, right? Because that would be the process, they're being transcribed, it's not going to be spliced, it's going to be spliced, it's going to be degraded. So here there's just more RNAs being made, so the cell is just activating its gene, it's downregulating it. And with this trick, you can then actually sort of learn this derivative here, and you can sort of start to potentially building things like that. What we've been doing in this follow-up, we've been extending this simple model for the estimation, simple model for estimation, this thing by a dynamic model, and been fitting that actually with the maximum likelihood and an MDM type of algorithm. We could find dynamic things because you have the likelihood available. We can actually find a shared trajectory as well, and we also had some robust implementation in our SGD. This was worked by Volker Björn, who has been really putting this strong effort in there. And that's one of the examples on this pancreatic endocrinogenesis example that I showed you before. So again, this is the stem cells. There's some cell cycle behavior that you see here. So each of those arrows— so each, I should say, each of those dots will be a cell. Those cells will be in 20,000 dimensions. And for each of the gene, you would know which direction it will be going. And then you just project it down to 2 dimensions, and you see these sort of overall behaviors. And then here you see the cyclic behavior. Here you see the differentiation behavior. You see sort of it goes in different branches, such as alpha, beta, and epsilon stuff. All right. But of course you cannot ask different questions to that. So you could, for example, ask what's the likelihood of cells that happens here? You know, how have these been generated in this? So what we typically, we just look at these plots, but you know, essentially this, you know, just looking at two dimensions just removes a lot of information. So this animosity has been around, but I think we could do more with it. So the question we asked, and in the story that I will tell you in a minute, is can we, instead of just analyzing clouds of points, could we analyze the whole vector field? And obviously there's all kinds of strange things of noisy vector fields in 20,000 dimensions, but maybe some robust things we can extract from it. So could we go from vector fields now, could we learn then build the fate mapping? So this is the idea of Marius, senior PhD in the lab. To really make use of this full vector field and then apply diffusions on that. And if you apply diffusion, then like really going along the vectors, you don't need to guess the random initial state. You can actually estimate it, right? You could estimate the terminal states. And once you can estimate those states, maybe you can also tell for each cell what the fraction is, whether it would be going to the left or right terminal state or something like that happens. So this is sort of a common picture that is done in molecular biology, in cell biology, where cells sort of living on this Warthington landscape, and then maybe they take directions and take left and right patterns on this manifold, and maybe we can actually learn this from this data. I should say, so this we implemented in a tool called CellRank. That's the idea of this thing. So, this is a collaboration with Dan LaPierre's lab from the Morris Sloan Kettering in New York. And it's about to come out. So we're happy about that. So what it does, it actually combines in this directionality of RNA velocity with transcriptomic similarity. And essentially what we do is we take a cell and then we look at the velocity error and we compare the velocity error, which is noisy and so on. We can actually put to the model within a k-nearest neighbor graph, the position where sort of close-by cells are, and sort of learn a transition matrix, a probability of transition that goes in this direction of this matrix. So essentially what we're learning is a big transition matrix. It's going to be non-symmetric, obviously, as you see here, right? So the idea— and that's why we call it SARANG, because it sort of imitates, I guess, Google's PageRank— we do know some type of essentially Markov chain run along this transition matrix. That's sort of the simple start. You know, you have your transition matrix here, and then you let many random walks run, and then you sort of see where they average at. And if you see where they end, you would be able to estimate initial, maybe metastable, and then also terminal states. And once you have this, because you see where they go through each cell, you can then also compute fate probabilities. And now in mathematical terms, and actually included these slides, Michaela, because I learned, you know, this is here, this is happening at the math department, so you're allowed to show one or two formulas. So I wanted to put that in here. You know what essentially we do here in this transition matrix, you know, we have sort of this directionality here, and what you would actually like to do is, you know, you take a KNN graph and then you just apply this type of transition matrix multiple times. You start at a point here, you sort of keep applying it. So essentially you start building this, this set of steps, and, you know, depending on where you start and how you wait, you would be sort of ending at different things. So the nice trick, because we know how random walks on big manifolds. We know how to deal with Markov chains, right? We don't actually need to simulate 100,000 of those. We can really calculate the whole ensemble of infinitely many random walks by just sort of pushing this forward and then calculating properties of this matrix. Ideally, what you would like to do is you want to see sort of where this is stable, which things don't change, something like an eigenvalue decomposition. Because it doesn't work here, right? Because remember, this thing is not symmetric. So instead of the shooting position, you have to do something sort of a little bit more complex. We do this crowd shoe method, but in any case, you sort of find this then with a technique that we adopted from molecular dynamics called generalized P1 cluster analysis that helps us identify those steady states. And Miles came up with a very robust and scalable implementation of that. So now that we have these steady states, what can we do with that? Well, we can now— now that we have the pseudotime, we know where to be going with this say, the state A and state B, and we can see which genes are particularly regulated up in this one state A and maybe in the state B. And then once we have this, we can then sort of maybe rank genes by whether they would be pushing towards the one or the other phase. Of course, we interpret as pushing, but of course, really we see whether they'll be up in that one or the other state. And that could be sort of hypothesis for cascading potential regulation. And this in fact actually works. So just as a short validation, without sort of going too much into the bio things, this a dataset from mouse embryonic fibroblasts that have been transdifferentiated towards endoderm progenitors. And in this case, you see something happening across time. This is from a Nature paper from Sam Morris' lab. And you know, you identify setups. And the last thing she had, so this technique to tag cells, whether they would be actually successful transdifferentiated or not. And some would end in a dead end and some would actually properly reprogram. So we have this set here that actually works. This this one does not. And now we just ran CellRank from this start here, and then we see the successful versus the dead-end clusters. And indeed, if you just run this, this, this VedoCyto, you see sort of arrows, but it's very hard to make sense of them because, you know, if you project things in two dimensions, it's like too much happening. So instead, we run this, the CellRank, and indeed we see that like our fate towards— fate probably towards successful reprogram is going to be super strongly enriched in this area, as you would have from this cluster. And to be a bit more quantitative about it, we can now build a predictor of cells at various time points whether they would be successful reprogramming or not based just on cell rate. And turns out that this predictor is doing not bad job already at day 12, AUC of 0.8, and sort of keeps going up until 0.95. So I think that's quite encouraging. So going back to this example of the pancreatic endocrine genesis, so to wrap this, we can now ask how those epsilon cells terminate. And, you know, indeed what we find if you apply this, we find those three different terminal states. And then we can look what the dominant fate in which region is, and we see sort of those here will be, will be pulled up here. So in fact, actually, so what looks like one big heap here is actually sort of three different fates that just because of the U-mapping happens a little bit squished together. Where do alpha and beta cells come from? Well, this is sort of the region of basin of attraction for alpha cells, and you're sort of different here on the beta cells. And so on. And you can also coarse-grain this and sort of put this on with cluster representations and so on. So there's a bunch of tools how you can visualize that. If you're interested in that, check out spellrate.org. But of course, all these things do have limitations. And if someone has ever tried to run RNA velocity, getting robust vector fields out actually not always works. There's quite a few situations where there's a whole bunch of limitations to that. We recently wrote— sorry for that— recently wrote a review on limitations and sort of how the web things would go. I encourage you to check it out if you're sort of interested more in the math, sort of the conceptual side of things here. But let me show this sort of the key example that everyone sort of tried this thing out and just didn't go. This is in hematopoiesis. This is actually my first link to, I think my first bigger link to Cambridge, where I've been working quite a lot with the Gutkin lab on hematopoiesis early on. It was really fun. And yeah, a lot of good memories, but unfortunately design agnostic didn't work on that setting. I think reason is that in parts there's, in those genes that actually drive differentiation, maybe too few splice sites to pick it up with this simple proxy. And maybe also timescales are different. In any case, you know, so if you just sort of put things in by cluster, and then you sort it by normal pseudotime approach, or a bit more clever pseudotime approach called Palatir from Dan Lapierre's lab, this works beautifully. So the hematopoietic stem cells are here and sort of goes down here towards the on the other side. But if you run our net velocity, you know, the arrows will actually look in a different direction. And that's the reason, because those footballs that you need to fit, those will be sort of going straight and just like don't work. All right, so what can you do? So one thing that we've been discussing quite a bit in the lab with Volker and now Philipp Weiler taking this up is how can we extend, and Marius of course in particular, how we can extend this. And we've been thinking about how to change the system that you currently have, which is take a kernel and put it in transition matrix, make an estimate, and get gene trends by sort of changing the set of ways how you calculate transition matrix. And one thing we've— so we have a bit of different set of estimators, but there we've been now adding also similarity as pseudotime, and then also other type of estimators. For example, the most recent one, lineage type of kernel, which we actually presented at a workshop, at the currently ongoing NeurIPS workshop. Where I heard from here that there's students jumping back and forth between these two. So in any case, you have these different options to do here. And let me just show you briefly what this one kernel is. And I think there's quite a few nice conceptual ideas, but maybe you could also build it again. I think there's a lot of people around here from, from quantum reasons. I think there's a bunch of more things that you could put in here to be better. So particularly this time series one, very interestingly connected to optimal transport. I want to speak about the pseudotime one though. So in this case, we just take the transcriptomic similarities as well as the pseudotime now, so not the very velocity but the pseudotime, and then we just sort of run it through this sort of weighting thing here. If this is sort of too strong negative, we just put it down, but if it points the right direction, we sort of accept it as a transition matrix. Just after normalization means we always sort of start going with a higher probability towards larger pseudotime and with very low one towards that's not the whole idea. That's like what we call the Palatier kernel because we just retook it from that. So if you now do this, so we take the Palatier pseudokernel, we add it to this KNN graph with this kernel as I showed before, and then we get arrows, right? Because we get now directionality for each of the cells. So where's the highest probability waypoints? We project them onto our 2D, and you look at that, lo and behold, the vector fields suddenly look nice. And now because the vector fields look nice, now you can do the cell rank analysis again. So you can look for the steady states of— so for the sort of attractive states of that thing. And so we can automatically detect all the right things like monocytes, CPs, dendritic cells, and so on. And we can also look at fate probabilities towards each of those. So now that you have vectors which are not RNA velocity, you know, these are sort of pseudobased, pseudobased ones, you can do this sort of So I think that's actually quite a nice opportunity about adding also multi-omics observations such as attack to it. I think that's interesting. So before now going to my second story, I just want to show that this has obviously also a bit more hands-on applications. So my main collaboration partner Heiko here from Munich together with the senior postdoc group leader Annika now, as well as Marin and Sophie, recently been moved on to different places. But we're looking here at the epithelial gut, where we've been essentially asking how those stem cells at the crypts are making all these different cell types so quickly. And the key question that Annika and Heiko asked, do we see direct differentiation, or do we see sort of this multi-state type of differentiation? This multi sort of hierarchical one, which often has been hypothesized in blood, or maybe this direct one. So when is the decision happening, essentially? And to answer this, we did single-cell RNA-seq. And then we get sort of this this cluster after we annotated and did a lot of work. I just showed one slide, but there was a lot of— particularly early times, we didn't have all these tools. But what you can now do is you can now sort of coarse grade it, look at the clusters and see how you get sort of the key transitions between things. Then we realized we have our intestinal stem cells here. And before we go into those terminal fates always out there, we always go through univariate progenitors. So what we found there is that this whole trajectory thing tells us we find only uniported ones. And indeed, we were able to validate this then also with directed expression. And as a follow-up, now we use this atlas for metabolic challenging study. This also just recently came out, where we now add sort of diets. We look at mice now on a high-fat diet. Again, took epithelial cells and now got this distribution of those. Here, the stem cells again, and some enterocytes and goblet cells here, and the control versus the high-fat diet. And here you see quite interesting effect. It's actually not that it's like sort of a big new cell type popping up, but it's like just a change in distribution. Essentially what we need to do is we need to do differential distribution comparison in 20,000 images. That's sort of a little bit hard. That's why we have clusters and you can sort of do tests. So essentially we just projected on these, but it's still interesting math in the sense that this is now censored, right? 'Cause they all integrate to one. So they all integrate to one. Actually a problem where you sort of need to deal with this additional constraint. Yeah, and we can be potential tools for that. But so we saw that there's differential happening, and then you can also add pseudotime and RNN velocity top of these things. So look at these particular differential issues. I will leave it here and come to my second topic. And I don't know, maybe, uh, no, don't take questions now, but like Right.

So yes, I think that's the best. We collect them at the end.

So, so this was now essentially one study or sort of using this other study for predicting your own disease or the challenge on top of it. But I think the future is that we will be doing these single-cell organ atlases in a more consistent fashion. And you might have heard about this, this project, the Human Cell Atlas, where essentially sort of trying to build this periodic system of of elements, but not of elements, but of cells. You sort of try to organize things, right? And if you ever had to look at genomics, early times you were sort of assembling DNA sequences. Nowadays you never do this. I never did that. You just like map onto existing ones. And I think this will be happening in our field as well. Once we have this atlas, we start getting this region, you never do all these sort of clustering annotations, all these manual things like learning images as you showed before yourself. This has been done by someone else, this is a put your stuff on top of it. I have one slide here how to actually design this type of things and like how to select the number of cells. I won't go to that, this has been worked by, by junior PI at the institute called Matthias Heinig, power design thing. I think it's really fun if you're planning your next experiment, let's look at how we can use these atlases and there's some recommendations actually how you should be dealing with that. Let me briefly, but let me go back to this atlas question. So You know, once we have such an atlas, what is actually first? What is it? Like in HTA, we wrote that it should be a collection of cell reference maps and characterize these setups. Yeah, I should say, you know, this is like the thing that Sarah and Aviv Regev has been setting up and contributing to that. It's really fun and fantastic international endeavor. So thanks, Sarah, for many others, of course. So if you just had a single dataset, you know, this is sort of how maybe cell type look like. That's what I showed you before. But in an atlas, we kind of want things to be robust, these things to be generalized across different maps. That's something that usually doesn't work as easy. So if you do this now for the studies, these things usually look like that. Obviously, this is an artist rendering, but, you know, what we actually want to do, we sort of do some type of integration that it looks like that, that sort of the same cell types are close by, independent across the studies. So what you need to do, you need to do some type of integration and normalization of potential machine, you'd call it domain adaptation. So how to do that? So for atlas building, we need robust data integration, essentially learn a latent space. That's usually what you work on, right? Like a KNN graph or some type of embedding with an autoencoder, as I'll show in a minute. But you need to have this joint latent space. And once you have that, you can then ask as a user, hey, where are my cells living? How can I project on top of them? We've been doing a bunch of benchmarks on how to do this data integration. But I think that's interesting questions. So we do, for example, contributing to building these atlases in mouse and humans and integrating this. And also, we've been using these atlases also, for example, this lung one that we had to now see how variation of cell types in human actually distributed. So this case, we've been taking first actually samples from the whole atlas, and check where entry genes for SARS-CoV-2 are localized and finding, for example, outside of the lung, also regions in the brain where this is active, and particularly in the bulbous olfactory, so where you do have some smell sense. But in the lung, and this study here, we actually had more than 280 samples across 160 donors, and we could look at variation across people where entry genes in, for example, alveolar cells, so these air intake cells were localized. And we saw this is enriched in men versus female, so male versus female, as well as in older people versus young, as well as in smoking. So there was sort of some type of meta-analysis that you could start doing. I think it's an interesting outlook where we could be going with these studies getting bigger and bigger. But I wanted to not stay on the example but speak a little bit how we learn these things. Thank you. So somehow learn latent spaces in a robust fashion. For this, we use architectures called— there's different things that we do. In our case, we use an architecture called autoencoder that essentially Neil Joshi's lab in mind has to sort of pioneer and bring it into the CSR field. And I guess you know how this works, right? So you take this high-dimensional dataset, sort of compress it, blow it up again, and make the output as simple as possible to the input. So you essentially just do representation learning, right? And, you know, for example, if you have sort of noisy gene-by-cell matrix here, then you sort of reconstruct then these type of trajectories here. And, you know, in this layer where things have been squeezed down, you know, in this sort of linear case, you know, everything here would be just like PCA, otherwise it's just more complex. Well, there you sort of maybe see this process, and this is sort of our sketch how it should be looking like. But indeed, this, yeah, this should maybe also encode something, but indeed it's actually true case. So for example, here, this was our first type of paper in this direction. We used it for denoising. It actually worked out well, but in this bottleneck layer, we saw that in this case you see PBMC cells. Each of those colors would be a different cluster where those clusters actually close by a range like you would do in a UMAP. Of these new plots. But in this case, it's literally just a 2-dimensional bottleneck that's being plotted. So this told me that we actually do learn an interesting latent space, and then we leveraged this ever since. So for example, in this follow-up here for something called— we call it SC-Gen, a generative model was building upon this idea of using variational autoencoders. So then we've been asking what happens if we sort of have a shift in this latent space by perturbation. So this was a This is just work by Göttingen Ersland who moved to the Broad and Luca Simon who moved to Texas. This was worked by Mouloud Falahi who's a senior PhD student. In this case now we see control cells, PDLCs, that are being stimulated across different cases and so it's just matching cell types. And we asked if we remove this one cell type can we reconstruct it? Actually turned out we were able to do so. So we can essentially model latent space structure in SNM. All right, what can we do with this? I mentioned this idea already that we shouldn't be— we should be using these atlases to map stuff on top. I think in machine learning terms, this is something what we could call transfer learning. So traditionally, we sort of have these different learning things, just do everything separate. But in transfer learning— sorry for this very simplified sketch— you would transfer knowledge, right? So this is a sketch from Raoul's and Tijah's first paper where they did this type of alignment Jorma Junis' lab had a very related one with Laleh Akverdi as first author. So this idea has been really pioneered by those two labs, and we've been thinking how we can introduce integration but not mapping it to a reference. So this is again a work by Mohit, which we call Single Cell Architectural Surgery. This will be coming out, I think, in January. So we have public reference datasets across a bunch of different studies. Then we do a pre-training of these reference models. In this case, you don't have a normal autoencoder, but we have a ratio conditional autoencoder where conditions are reference samples. And then sort of the key trick to sort of make transfer, we don't actually share any data, we just share the encodings here. Because if you share the encoder, you could give it to another person, they could like reconstruct the latent space without having the original datasets, right? And then you can use this to transfer, for example, between healthy and disease loss studies and so on. So if you now have query data, what you would be doing is, you know, modify the architecture of the thing. So you add a bunch of additional labels in the reference. That's how we call it architectural surgery because you're going to change the network. So you add a bunch of links to the next layer. We tried doing it for the whole thing, but profiling showed that in our case, in many cases, actually just having this to the first layer was sufficient with respect to the construction quality. And then we were able to reconstruct and sort of adapt this, sort of modify this network that this new dataset fits in. And you could do it for multiple adapters and you could also upload this. So this was actually working out. So just show you one example here. In this case, again, we look at a COVID-19 dataset of people both in a healthy as well as in a moderate and a severe situation. We first build a reference of pulmonary lung PBMCs. This reference dataset, then we map lung lavage data from those patients on top of it. And first you see that This language actually nicely is being able to map with SCRTs. The moderate ones would be clustering here, the severe ones here, and sort of all of them up here. And because now we have an atlas, we can actually see what's going on, right? So we see, for example, here this would be macrophages, and those macrophages localize in different places. And if you drill in, then you see that some of them, I think in the moderate case, the tissue-related ones, whereas you want more associated ones that you can sort of validate this. You can also look into CD8 positive T cells. I'll skip this example. If you want to just check it out, this recently came out. But I think what's nice that we've been sort of extending this to actually include also multiple modalities. And I mentioned this was sort of a multimodal version. And this is something I wanted to show here. If we take now a single-cell architecture surgery, you can actually do this also for multimodal observations. So not only RNA, but RNA and state protein, RNA and ATAC. In this case, we've been using as a base model that you then modify with the surgery something called TOPII from Nir's lab to build a joint model of RNA and protein. And then we take a new curate model that only is RNA that we then ask to sort of include into this one via our shared encoder in this scRNA-omics approach. So then we integrate them, then we have reference and curate together, and then you can see if those curate data, we can now impute the protein side, whether they make sense. Indeed, they're sort of nice and localized for for the different CD3, CD4, and CD8 cells. So it was quite exciting for us. And now we're extending this to make it more into a classification. And maybe to show this one point, I know that Michaelis, and I'm sure many others of you are interested in sort of this direction of more explainability in AI. So we've been thinking how to understand those atlas in a bit more sort of more, more, more, more, more captured in a better fashion. So we have here again our VAE setting, right? But we want to constrain it towards— by adding prior knowledge. For example, taking something called gene programs, which in many cases are just some bags of genes. And we kind of want to say, hey, can you explain our latent space in terms of these things? We've been doing some previous papers— I actually haven't included this— at some ICML and ICMP workshops where we I think at least in ICML we had stuff doing this, doing some type of latent space disentanglement, but we found some interesting things. But in this case, we really just sort of added into the learning. So we do a regularization of the encoder, of the decoder, where we actually have a fully nonlinear encoder, but we just have a linear constraint decoder constrained by this regularization, for example, masking of those gene programs, and then adding also additional lasso type of layer here, actually group lasso layer, where you only allow to select the variables that are actually being used across the whole set of gene programs that could be present in the product. In this case, we can actually then learn potentially connected gene programs and can sort of understand more what's happening in this latent space without first doing like all these differential tests and so on and potentially sort of losing information, having really strong following. The nice trick now that we learn such an atlas, again, we can do the transfer learning thing as we did with SCR. So if you now sort of map, now you have such a nice atlas, you can now map your reference on top of that. But now you can ask if you have now a new QA dataset, there might be additional pathways there. So we actually add another thing of surgery now in the decoder to allow for unknown pathways. So that's these two steps. First, we've regularized the decoder and then make surgery on the decoders more. That's the idea of this approach. Let me show what you get with this. So first, and that's I think really promising to me, we were actually able to show that we don't lose expressiveness versus sort of nonlinear decoders. So in this case, we look for example in blood and compare this to a fully linear model. It actually looks as if you just calculated a bunch of metrics overall, it sort of looks pretty much similar. And we do this now across also 12 different datasets in colon, liver, lung, and heart. Hard. In many of these cases, we kind of see similar, maybe better performance of this regularized extreme versus a fully nonlinear decoder. And overall, across these datasets, we actually, in most of the cases, this constrained aspect of this programmable mapper here is actually improving on it. So what do we do with this? Let me just show you this one example. So in this case, we have a bunch of datasets in PLAT with a particular activation one here where— so this would be the reference, the red one is now the stimulated one. This is PBMC cells stimulated with interferon beta stimulus. And what you can now do is you can now say, look in the latent space and see which factors are sort of highest up in which direction. So for example, what type of latent space factors are showing up there? And then you see, oh, it's interferon signaling, which is sort of what you would expect, but you know, you find it automatically. And we can look now at the gene programs, whether they would enriched. And we see that these two signaling things are actually up in different cell types. And then we can actually see what happens if we remove those gene programs in this case. So for example, if we remove those 5 top gene programs, suddenly this perturbation answer is not as strong, it actually goes away. So this actually tells us that this group blast was really selecting the correct programs. And you know, if you can compare with other tools, it doesn't work as well. I skipped that part as well. But then this is the last part, because we have this search, we can actually try to find those new programs. So in this case, we now remove, for example, B cells and related pathways such as those a little bit, and we challenge the model: can it find this now during a new query that actually has these terms? So in the new query, this thing actually tries to build new terms, and we add a Hilbert-Schmidt independence type of criterion to actually make sure that those terms are not disentangled with each other and also with the rest. And we then can analyze what sort of up in those different terms. We actually then see If we plot, for example, program APC versus myeloid, then we see that actually picks up correct B cells. What we're now doing, we're now transferring also learnable embeddings to COVID samples and we go in different directions. So I'm nearly done with my time, so let me just spend 1 minute on an output with perturbations and then we'll wrap up. So, well, actually I think we started 5 minutes later, so Mihaly, you need to tell me. I can wrap up now.

We have the time, we have the time, Fabian. Thank you.

Thanks so much. So thank you for— sorry, I— so yeah, I'm so bad.

We are eager to learn as much as we can from you, so please, please finish the wonderful talk, and I'm sure there are many questions.

Thank you so much. So this last thing, so this was a bit about interpretability by adding priors. This other thing that we've been trying to go towards interpretability was to have a bunch of perturbations happening and see which directions they would be pointing. We really want to understand the effect of perturbation. And there have been some approaches for doing so. It's actually early on, this was a paper from a postdoc of mine working together with John Marioni from EBI at Ollie Steketee at the time, from EBI, sort of removing the cell cycle contributions. And this is also from Aviv, where she's been trying to build factor models. In this case, you know, once we have a factor model, we can sort of understand where things happening. But this doesn't so much work in our case because we have these strong nonlinearities. So how can we bring these things together, and how can we do this at scale? And I think we need to do perturbation modeling at scale because, I mean, at the moment things are still a bit expensive, but things are being multiplied complex like crazy. And, you know, if you want to do a drug screen or something like that, you may say with 100 drugs you want to look at combinatorics of those. This gets already expensive. There's something you cannot do experimentally, in particular if you do like very many numbers of perturbations. So this case, we thought it's a good idea to team up with people who are good at building complex models. In this case, Anna Klimovskaya developed this path from Facebook AI, where we've been putting together what we call the compositional autoencoder. So the idea is we combine these linear models that we had before with flexibility of deep representation. It's like a factor model in latent space, essentially. The idea here is that we— this is work by Mo together with Hugo, Carlo, and Nachik from my side, together with Ollie Dyer from Facebook AI— where we now take our encoder-decoder type of model in the variational version. But now add the perturbations. And typically, so perturbations often we just added discrete conditions there, but then you don't learn actually abstraction between those. So instead we add them as factors. We just build a factor model in latent space, and those directions of those factors we just learn during the whole learning of the weights of the decoder and encoder, just like shared across all samples, right? And to make sure that the z does not contain already information about the drugs, we add an adversarial loss discriminator to remove information about those perturbations before. So this way we can actually learn latent spaces, we can predict novel conditions, we can do dose-response modeling, and potentially also just look at this interesting latent space. And I have two examples for that, and I think I just show you this one here where we had actually a huge dataset from Coldtrap, now three cancer cell lines, nearly 200 drugs, 4 doses, a huge thing. Can you imagine 4,000 perturbations, very noisy, but half a million cells. You see some structure here, but what we did, we now fit our model, we have an out-of-distribution, so we left out this Burkina stop in this particular cell in MCF7. And we see that our prediction actually sort of coincides quite well with the true one. This does actually happen systematically also, so we get very good prediction across all the different genes, even the differentially expressed ones.. And we sort of understand structure. We can plot latent space, and that's sort of the nicest thing in latent space. Like, those factors actually similar sort of in terms of structure actually cluster up similarly as well. So that's sort of, I think, an exciting direction where you could go forward. I would leave it here. Of course, I say I would leave this example here. I say space is becoming a big thing, and for this we've been sort of coming up with toolboxes as well. Check that out. I think it's called SquidPy if you're ever interested in spatial settings. A lot of fun things to do. Now let me wrap. I spoke about super-temporal ordering that essentially learned one-dimensional latent spaces. I spoke about this single-cell velocity and RNA velocity and cell rank to learn Markov chains there. Spoke about data integration and single-cell architecture surgery to reference learning on data integration. And then I spoke about this compositional perturbation, autoencoder perturbations. We have a bunch of extensions. I think it's very exciting what's happening at the moment with respect to spatial information. So you kind of want to learn from these things, cell-to-cell communication. One thing that's been applied from Sarah's lab is just look at it at projected ligand interactions and thereby sort of learn in these disassociated datasets where there's sort of similar things happening, similar contacts between cell types. And spatially, I think we can can do more. And that's sort of very exciting data also actually coming out of Cambridge. But then we also want to interpret those data in spaces, and we've been proposing some ideas, but I'm sure that's much more to come. Yeah, and I wanted to highlight these open problems in single-cell analysis. We have a website, openproblems.bio, and we're particularly running there also a NERBS competition, multimodal single-cell integration. We have the largest multimodal dataset around across 5 different labs, many cells, a lot of manual annotation, a lot of work actually running this. Malte Lücke, particularly from my lab, has been doing a tremendous job with Chris and many others there. And yeah, this has been, I think, really, really fun for us in education also to run things. So we will speak about that on Friday. This is the team. This is how Bavaria always looks. Please come by. Thank you again for the invitation. It was fun talking. I think I acknowledged already everyone, and I'm looking forward to discussion.

Thank you very much, Fabian, for this wonderful talk. What a tour de force. I'm sure there are many questions. I have quite a lot myself, but maybe we can open for discussion. So please raise your hand if you have a question. Yeah, and applause from everyone, as you can see. Uh, it's unfortunate that we cannot hear all of them, but So, um, while people are starting to ask questions— so please, um, raise your hand or put a question in the chat that we will call on you— while people are accumulating questions, I want to ask a more general question, Fabian, since you are one that is capable to really, um, combine cutting-edge biology, bioscience with cutting-edge machine learning. For the ones among us that do not understand enough about biology, what do you think are good problems for machine learners? Where do you think the opportunities are? I saw a lot of work on autoencoders, and we do know that they are a very important piece. Seems of the puzzle, but do you see other opportunities and other areas where you think that machine learning can contribute and develop tools that you would find useful to bring to your research agenda?

Yeah, thank you for asking this. I think that you're really hitting at a very important point. I've been sort of trying to bring in a lot of machine learning examples into this thing, but I think like there's so much domain knowledge needed to actually really make good examples and sort of make sure that whatever you do is actually relevant. That is hard. In my former life as a PhD and early postdoc, I was mostly doing sort of more method. Only paper I wrote, I literally had the same example like in 5 papers or so at the end because I didn't really care that much. So this has been changed a little bit, but I think it's going to be hard to bring these languages together. That's why we started this openproblems.bio thing. We've been trying to simplify and sort of boil down these questions and also the access to datasets and the complexity in such a way that, let's say, as a starting student without much interest in a particular domain but maybe wanting to do just integration or joint latent space like deep representation learning, essentially you could go ahead. So having said that, yes, I think one of the big questions in Aufried is not, you know, in Aufried many things are unsupervised. Because you don't really have a real label for it. You do these big things, but you don't actually care. You want to see how the whole thing is. A lot of multi-instance learning questions, but I think the biggest one is representation learning. And the representations are really bad. I mean, we know in computer vision, you don't really use autoencoders anymore. In our case, because the GxG things, there's this complicated regulatory structure, but by itself, You often deal with this as if it was tabular and it's like a bit too limited. So I think adding structure there, I think, is still one of the unsolved systems. There's a few papers coming out from time to time, just like trying additional things. I think recently I saw this SEBERT on bioRxiv. So yeah, you try whatever you have. But then also adding time structure. You know, we usually destroy sound in contrast to your patient data where you have trajectories, right? In our case, trajectories are always just distribution matched. So you need to do large-scale distribution comparisons. For example, Wasserstein distances is a common thing to transport across those things, and I think there's interesting problems there. Yeah, and then the latest one— so this was sort of all on the synthesizer domain— I think the latest one is that of course combining this with medical information, like with patient phenotyping and so on. Because, you know, you saw this in this one study for COVID, we had an example of across 160 patients. So I think it will become common to do like the whole immunology, which is obviously en vogue very much now, also on the single cell genomics level. How do you bring these two levels of heterogeneity together? And I think for this some type of multi-scale sort of approaches could be— but that's just sort of a few. I think it might be a good idea maybe to write some type of small perspective on, you know, what are particular machine learning problems in different areas. That might be something that could come out of our LS Health program.

I definitely hope so. I have many more questions, but it's time to kind of allow others to ask questions. So Elias, Elias Heydari, would you like to unmute yourself and ask the question?

Yeah, so super exciting talk as always. I had a very general question and I always wanted to ask you, but What do you think are the biggest challenges in mechanistic modeling of biological systems like SCiVelo or CellRank, which specifically use machine learning, particularly when it comes to validating such models in the lack of ground truth? Thanks, Elias. Thanks for bringing this up. And good to see you. Well, actually, I don't see you, but good to hear you. So this, I think, and thanks for bringing this up earlier when you asked general problems. I think he has had one more. You know, often we have a lot of prior and we actually have a lot of sort of smaller scale mechanistic models. How do you bring those two together? Maybe population models, but maybe also these regulatory ones. And I think, yeah, unless you also want to call it mechanistic modeling combined with machine learning, I think it's really hard. And, you know, at the core, I kind of really try to— I would love to solve how gene regulation works, because if I know that, then I can know how drugs perturb, how, what goes wrong in disease. It's like sort of, I guess, one of the biggest holy grails in our field, right? And obviously I won't, I was like doing things on that, but I think one of the challenges that I really have is that the models where they really work, the mechanistic ones, either very simple model systems or very reduced ones like one gene, just let's say one broker, like really simple yeast cell, but like simpler yeast cell, like like, you know, all these complex multicellular organs that we have that are not only made out of cells but communicating cells that then form tissue that then forms sort of— because there are big vessels going in for— from whole organs. It's like really complicated. So the whole multi-scale nature of things really gets hard. So I think multi-scale is one of the biggest unsolved things, and I think we have to take a step away from trying to be able to model everything mechanistically, but in part just replace it by more black box-ish type of models, but where we can, we should model mechanistically to say, if we eat something and sort of we have things going somewhere, or like the blood system, it's like a simple compartment model. For this, we don't need now a big machine learning model. So I think trying to merge these two, I think is something that's very exciting. And here within Helmholtz AI, we actually have people coming in from matter, from physics and so on. And they all have often some type of mechanistic aspect to the data, but then your data gets more noisy. So I think bringing together this world of simulation-based systems and here these sort of more data-based ones, I think it's a little bit still untouched, at least more to be done.

Thank you, Fabian. Benjamin, would you like to ask the question?

Hi, Fabian. So I was curious whether there are experimental pipeline developments that are coming through that you're particularly excited about? I mean, in respect of, in respect of these kinds of approaches. So, you know, I'm thinking of, of things like, you know, what Michael Alevitz is doing on being able to trace and profiling and so on. Do you think that there's technology which is going to be transformative in the way that you can start addressing these kinds of questions about lineage relationships and self-made choice? Yeah, that's a good point, right? I mean, this was sort of one of these things where you actually follow them up single cells and build images on, and this is actually starting to happen. So there's been a bunch of these approaches 2 or 3 years ago, and I think it was even, it was like highlighted to somewhere really highly where you have, I think methods called SCAR trace or something like that, where I'm a bit naive there, but essentially combine some kind of clever CRISPR technique that sort of randomly generates these barcodes and then they're being inherited, right? And it kind of works very well in systems that sort of were very quick at producing things such as zebrafish, for example. Alfred Juncker has a nice paper on that, Lineus, I think it's called, and there's some others. And there's been another set of direction from Aviv's lab, where they've been using mitochondria and somatic mutations there to then also trace. But in any case, there's some clever ideas that people have, and I'm sure that I think Eichhorn Ullmann has also something where they can then follow up also a little bit progeny, not very deep, but a little bit. And yes, you're right. And of course now what you could do, you could now add this to, for example, this transition kernel that I was showing before. And in fact, with, with, um, um, um, Moini Tsan's lab in Tel Aviv, who Marius recently visited for lab exchange, actually in this current challenging time, we have a joint thing called lineage, sort of a follow-up to this lineage cell rank type of thing, where we can, I think, include this constraint also in meta-transition matrices. I don't think this will be super much ready yet for, let's say, human systems where you should maybe do genetic engineering, right? But you should definitely not, okay, let me be explicit. But I think this will be cool for validations and then be more explicit. And then of course, you add multi-scale, so now like multi-omics, now you maybe have also attack or so, and you can include more regulations. So I think there's interesting data coming soon.

Thank you. Thank you, Fabian. Jake Taylor King has a question. Are you here, Jake?

Oh no, yeah, uh, my, my question was actually, uh, the one that was asked earlier on technologies you're excited by. I just wanted to chase up what was the reference you were talking about when you said that the velocito ran backwards? Oh yeah, I mean initially like when RNA velocity came out, so I think a bunch of us tried that and sort of was kind of common knowledge, hey, this doesn't work work everywhere. And then we had this, we had, I think it was, yeah, I just put in the chat, Biagnett Allmolecular Systems Biology, Bergen, Geiss, Molsys Bio, I think 2021 or something like that. But we've been discussing all kinds of caveats of things, and there we sort of brought this limitation in there as well. Yeah, but if you sort of want to I'm happy to send this around as well. Thank you.

Thank you. Mo?

Mo, are you here? Yes, can you hear me? Yes.

Yes, thank you for your talk, Professor Thites. My question is really regarding analyzing multimodal data. There's been some developments recently and I know it was like SOHA, they've come up with a multimodal aspect. My question was really regarding clustering and studying these cells, because when you cluster these cells based only on either protein or the transcriptomic modality, you get different visualizations. And it's also different, say, if you use total VI or MOFA. My question was really regarding In terms of using more modalities, how would you decide which modality is going to be a priority in inverted commas in terms of organizing the data to allow us to really extract something meaningful?

That's a good question. For me, I usually like taking a metaphor. In this case, let's say we would be actually sitting in an auditorium, right? And then you would be able to quantify different stuff of us, you know, like, I mean, obvious things such as height, gender, age, things like that, right? And ideally, if you really want to get similarity of people, you like take whatever you can get, make your vector, calculate some distance in a cluster or something like that, right? And that would give you, I guess, when you might call it unbiased, but obviously it's biased because that's just going to be things that should be quantified. And I guess that's a little bit how people see multimodal at the moment. Of course, there's a bit of a danger there, right? There could be one of the direct spaces has something like 100,000 observations, the other one has like 20, and then things might go imbalanced. So there's a whole bunch of things where people correct and so on. But I think in the end, it always depends on the question. So if you really want to do completely exploratory analysis, analysis, I would really recommend to maybe not do all that much and just like take the joint ligament space and do it on that. If you're interested in immunological question, I think it's often very important to add protein there because like subsets of T cells are usually often just very well identifiable using surface markers. And in other cases, you know, maybe a single-cell attack helps. I don't think there's so many convincing examples where single-cell attack actually is improving, but I think I think this will come. So yeah, I guess it really depends on the question. And maybe there's some sort of attribution type of idea around it as well. So depending on the question, you can say to some type of attention, like in autoencoder, you can add attention and you can say which model is actually adding the most. If you're interested in this, we had a paper, I guess it was also some, maybe it was an ICML workshop or so, called Multi-Code Might integrate, might integrate. That's a very clever name you put there. It was just an abstract, but then we can sort of discuss a little bit how this could be done.

Thank you, Fabian. Maybe time for a final question, and that's from one of our own PhD students in the center, one of the new recruits. So Alex, Alex Norcliffe, would you like to ask your question?

Are you here? He's writing about neural ODEs, I see in the chat.

Yes, yes. So maybe I can ask his question. Both he and our groups are working on that. So the question was, do you think a latent neural ODE might be an interesting way to model these gene regulatory networks?

Yeah, I like that a lot. So we've been playing around with this quite a bit, like spending like a year on that particular one post in my lab. And we have some results, but it turns out that getting this robust on this complexity data, we have not been as successful really. But maybe we're also doing something methodologically wrong. So if you have people—

maybe an opportunity to collaborate with us.

Wonderful.

Thank you so very much, Fabian, for such a wonderful talk and so many discussions. You, you are very inspiring to us. We really hope ourselves in the center to straddle these different worlds and bring them together and make a difference in both. So thank you very much, and we look forward to seeing your NeurIPS endeavor and what these new problems are going to lead to. So maybe if you send us a few information about that, we can distribute transmit it through all our channels to everyone interested. So yeah, so please, please very much do. Thank you very much again. I guess a round of applause from all of us again, and thank you for visiting us.

Wonderful. Thanks. Good to see you. Thanks, Ben, for joining as well.

Great seeing you. And Fabian, we have a follow-up discussion with you. I will send to you the Zoom link, um, in the email. Maybe that's easier.

All right, see you in a minute.

Thank you.

Thank you. Bye.