Scientist Stories: Karsten Borgwardt, Machine Learning in Medicine & the Early Recognition of Sepsis

Yeah, good morning everyone. This is the data science seminar today with our speaker Carsten Borgwardt. Before we start, there is a short introduction. First of all, please mute yourself, and after the talk there is the opportunity to ask your questions, and you can do this directly to Carsten. So at the moment, I will hand over to Lena to introduce Carsten.

Yeah, it's my very great pleasure to have Kasten today here in the seminar. He's a full professor of data mining at ETH Zurich, actually in the Department of Biosystems located in Basel. He was quite umtriebig in the most positive sense academically, studied biology in Oxford, computer science in Munich, then did his PhD also in Munich. And as a postdoc and later group leader and associate professor, he had positions in Cambridge, Tübingen, and finally Zurich. I probably missed something, but quite a lot. He's also a fellow of Alice Health and Alice faculty member, and he has won numerous very prestigious awards and grants. I cannot list them all, but maybe for the Germans among us, it's interesting that the German news magazine Focus included him in a, in a list of 25 individuals who could shape the next 25 years. So this was in 2018. And I actually remember when I first came across Carsten's name, this was, I don't know, many years back actually, and I read an article about him because he won some prestigious grants, and I remember him saying something like Well, you know, I was just with the right topic at the right location at the right time. Very modest. That impressed me a lot at that time. And I can confirm that he has no star allure from my perspective. So it's really a pleasure working with him. And of course, his research is excellent, which is the main reason we invited him here today. So yeah, the stage is yours, Carsten.

Thank you very much for the kind introduction and also the kind invitation to speak in this seminar. Yeah, I remember the statement. I was a bit inspired by a definition of personalized medicine that Barack Obama had given. He had said that personalized medicine is giving the right treatment to the right person at the right time. So maybe this is the source of my statement in a way, at least stylistically it's like that. Good, so welcome everyone to my seminar. I'm really happy that there's such a large audience here. I'll go to full screen mode. Yeah, so what you can see here on this slide is not only the title of my talk, Machine Learning in Medicine: Early Recognition of Sepsis, which I'll talk about in the next 45 minutes. It's also a picture of the new department that ETH Zurich is building in Basel. So this will be the Department of Biosystems in Basel. Maybe some of you know Basel a bit. It's not that far from Heidelberg. This will be next to the Kinderspital, the children's hospital, and to the University Hospital Basel. So we are looking forward to moving to this new building in 2 years from now. That's also the background you can see here on Zoom. So when you think about, when you hear about machine learning in medicine and artificial intelligence in medicine, there's a lot of hype surrounding this topic at the moment. So I usually start my talks by trying to take away the hype and define precisely what we mean by machine learning in medicine and how machine learning can contribute to medicine. And to start from the very beginning, let's first define what I mean by machine learning in the following. So machine learning, tries to detect statistical dependencies in large datasets. It's in that definition and in that sense also a daughter of applied statistics and of computer science. As applied statistics, it is looking for statistical dependencies in real-world datasets. As computer science, it is trying to develop algorithms that can handle the large datasets that are, that are the basis for this search for statistical dependencies. And now, and I think I can keep this part of the talk in this audience rather short, the wealth of health data that is now being collected all over the world in hospitals and research institutions provides a fantastic application domain for searching for these statistical dependencies. Because the statistical dependencies in these health data sets can have a concrete meaning. They can contribute to a concrete goal, namely the goal to personalize medicine, to find out which properties of patients correlate with particular phenotypes of that patient or the future health development, the future health trajectory of that patient. So we hope to discover statistical dependencies in these health data sets that allow us to improve diagnosis, prognosis, and therapy decisions tailored to the properties of each patient. And what I mentioned at the very, very beginning, Barack Obama said, It's all about delivering the right treatments to the right person at the right time. And machine learning is the tool that should or that could get us there. That's the connection between machine learning and personalized medicine. Now, if you look at what's currently going on in terms of research in machine learning in medicine, then I think that 3 clusters of topic, 3 key topics clearly stand out. The first one, and I think also Lena's group is very interested in this in the realm of surgery, is the automation of diagnoses and treatment.. So in this first cluster, we try to build algorithms that can mimic what a doctor in the clinic can already do. I have here an example paper where, in which the authors build an algorithm that can count metastases in slides from women with breast cancer, in lymph node slides from women with breast cancer. That's a task that's usually performed by a human expert. And you need years of experience for doing this properly. And in this work, the authors proposed like a computer vision algorithm that tries to do the same. And in fact, already 3 years back, this algorithm was able to outperform all human doctors except for a human doctor with many years of experience and without any time constraints. So it could beat this experienced doctor if the experienced doctor had to operate on a limited time budget, but not if the doctor had unlimited time. 3 years later, maybe the algorithm has improved enough to even reach this performance. But this is a prime example of where we try to automate the diagnosis, the expert knowledge of a human doctor in a machine, in an algorithm. Another example, actually quite famous example by now, is this artificial intelligence clinician by the groups of Anthony Gordon and Aldo Faisal, that learns the optimal treatment strategies for sepsis in intensive care. So based on intensive care records, this reinforcement learning approach tries to come up with the optimal sequence of treatments that should be given to a patient that is suffering from sepsis. We'll learn more about sepsis in the remainder of this talk. And a third Another example is by Piers Keane's lab. So Piers Keane here and his group proposed a scheme in which doctors can train an algorithm to distinguish images that show healthy subjects and that show diseased subjects without the doctor having much coding experience. So the whole system should be so automated that the doctor just has to classify certain images in the training dataset as being examples for cases, others as being examples for controls, and then everything else, the computation of the model, the data management, and so on, should be handled by this automated system. So this is their vision of where all of this can go, that a doctor can, even without coding experience, interact with such a machine learning system and make it a supportive tool in, in the clinical day-to-day work. Now, a second big topic in machine learning and medicine is biomarker discovery. It's different from automation of diagnosis and treatment in the following sense. In automation, of diagnosis and treatment, we are trying to mimic what human experts can already do. This doesn't mean that we can't outperform them with machine learning algorithms. If you combine the knowledge of the best human experts in the world in a machine, the machine may still outperform each of them individually. But still, we are trying to mimic something that already exists in terms of human experience or expertise. In biomarker discovery, we are trying to, to discover new biomedical knowledge. We are trying to discover connections that are unknown so far. And one example for that is the work from DeepMind, Tomashev et al., where they try to predict whether a patient will within the next 48 hours develop kidney, acute kidney injury based on the entire health record of that patient. And that's something that a human expert could not do to the same degree. A human expert could not handle this big data and this enormous volume of data that's available in the electronic health record and make a joint prediction based on that. So here, as an example, DeepMind is looking for biomarkers for imminent kidney injury. And the third big cluster within machine learning medicine is the whole field of biomedical data management. And this I mean in the broadest possible sense, reaching from the assembly of a genome to the administration, the management of the electronic health records in an entire country. This is a topic that is also of great commercial value. So an example from the town where my department is located, from Basel, is that Roche bought 2 years ago a startup, in fact Flatiron Health, for $2 billion. And Flatiron Health structured records of cancer patients' treatments in a database. They had noticed that cancer patient treatment is only in a small percentage of all cases recorded in a structured and uniform fashion. And they created a structured database for that. And after a few years, after support from Google and so on, this company, this idea, if you want, was worth $2 billion to Roche. I think that says a lot about the importance of biomedical data management in all of its details and aspects. Now, what I'm going to talk about today are algorithms that we try— that we develop in order to predict sepsis early. It's an instance of this field of biomarker discovery. As you'll see in a minute, our ability to predict sepsis is quite limited so far, although sepsis is a very important syndrome and a major cause of death around the world. So this will be the focus of my, of the remainder of my talk today. As you see here, I have 66 slides. I won't cover them all today. It's in fact, I'm presenting a tutorial here. I'll put emphasis on certain aspects that I hope will be most relevant to this audience. So why do we need biomarkers in sepsis. Sepsis arises when the body's response to an infection injures its own tissues and organs, and this then often results even in death. So it's the major cause of death in intensive care units around the world. The source of sepsis are numerous. Pneumonia is a very prominent cause for sepsis, and you may also have heard that people who die from COVID often actually die from a sepsis that is induced by COVID. So various organs can be the starting point of a sepsis, but pneumonia is a primary condition that, that can result in sepsis. And there are various symptoms of sepsis, for instance, severe breathlessness, passing no urine all day, but none of them is strong enough to be useful as an early predictor of someone getting sepsis in the near future. Sepsis has various stages, so we speak of sepsis, septic shock, multi-organ failure, which are increasing intensities of having sepsis, and the ultimate state is then that you die from this syndrome. So multi-organ failure, the failure of 2 or more organs, has usually drastic consequences. And even if you recover from sepsis, there are post-sepsis symptoms, so you're not fully healthy again. You may have difficulty sleeping, poor memory, fatigue, anxiety, and so on, months and years after suffering from sepsis. Also, it's more likely that you get sepsis again if you have had it before. The official international definition of sepsis is the so-called Sepsis-3 definition. Sepsis is a life-threatening organ dysfunction caused by a dysregulated host response to infection. Now, the reason why it's so important to detect sepsis early, to recognize sepsis early, as the title of my talk says, is that the identification of a bacterial species in blood still takes between 1 and 2 days after you've taken the blood sample. And from onset of sepsis, each hour of delayed effective antibiotic treatment increases mortality. For the computer scientists among us or for the non-medical researchers, the point is the following: sepsis is a form of organ dysfunction that results in an organ damage. So the earlier you can recognize this and take countermeasures in form of antibiotics, for example, the more you can reduce the organ damage that is being, um, that is being caused here. And the organ damage is then what leads to mortality and to these long-term consequences of sepsis, even if you survive a septic episode. So key is recognizing sepsis early and acting upon it. Detecting and treating sepsis earlier and better identifying high-risk subgroups is of highest clinical impact, as it says here. And we are now, as machine learners, trying to answer the question, can we answer this question using tools from machine learning and data mining, or can we even develop machine learning and data mining tools that answer this question? To describe it algorithmically, what we in our first studies on this topic put into the algorithm are the ICU, intensive care unit, data that describe the health state of a patient. This can be the temperature, heart rate, blood pressure, respiratory rate, oxygen levels, and so on, that are continuously monitored in an ICU. So think of a multi-dimensional time series that we are observing here for each patient, plus some lab tests that are being performed. And then the output, so the phenotype that we try to predict with machine learning, Is the onset of sepsis? Will sepsis occur within a certain time window in the future? Will this person suffer from septic shock or even die from sepsis? These would be typical output variables, typical endpoints that we try to predict using machine learning. And this kind of ICU data is increasingly becoming available around the world. In fact, in a collaborative effort, We also contributed to one of the intensive care unit datasets that's now being available on the central repository PhysioNet. So we contributed to the HiRID dataset, a collection of ICU data from Bern. Good. And with this input and output now defined, you can, from a machine learning or data mining point of view, take two perspectives on tackling this problem, or at least two perspectives we have taken so far. The first one is that we do something similar to an association mapping effort in genetics. We try to find episodes in these time series, time windows in these time series, whose trajectory is associated with the occurrence of a future endpoint, for instance, sepsis onset or shock or mortality. Second perspective is maybe more classic, the more classic machine learning approach, where we take all the information that we have on a patient up until time point t, and then we classify this time series into will develop sepsis or will not develop sepsis. So we treat it as a as a binary classification problem. And I'll mention our efforts or describe our efforts along these two perspectives in the following. I'll start with the first one because it's the less standard approach to, to the problem: association mapping in clinical time series. What do we mean by that? So I visualize here a one-dimensional time series with an event here described by this dot, this dashed red line. So let's assume that at this red line here, sepsis occurs. Now the question in time series-based association mapping would be, are there, before sepsis occurs, certain patterns, certain trajectories in the time series that describe the patients that tend to occur primarily before sepsis starts. So are there trajectories that are associated with developing sepsis in the near future? So this could be, as an example, such a spike in the blood pressure or such a, such a heavy variability here in the heart rate or this deep spike here with a sudden increase in the respiratory rate. So are there such patterns that are associated with developing sepsis? That's the question that we are asking in this association mapping on time series. And the idea would be that in those patients where that do not develop sepsis, these patterns are absent, cannot be found. So this is visualized here on this slide. Again, in a toy example, we have 3 patients blue that develop sepsis, 3 patients red that do not develop sepsis. I've highlighted here such a small trajectory like a double peak here that occurs in all 3 patients that later on develop sepsis. And in association mapping in time series, we are exactly interested in these patterns. In the data mining literature, these small trajectories, time windows, are referred to as shapelets. They are short time series that are associated with a similarity threshold theta. Why do we need this similarity threshold theta? Because none of these, these trajectories will be exactly identical between two patients.. So we need a criterion that allows us to tell whether the same trajectory has occurred in another patient, and we do so by computing the distance between trajectories from two different patients and then checking whether their distance is less than this threshold theta. So any time series with a subsequence s' whose distance to this shapelet s is less than theta contains this feature, this shapelet feature as it is called. And to state our goal precisely, our goal is to finding shapelets that are enriched in one class of time series, in particular in the class of patients developing sepsis. This term shapelet, as I mentioned before, is an established term in the data mining community, originally proposed by Ye and Keough in 2009. It's again shown here that we have this two-class setup, y 1 on the left, y 0 on the right, and we want to find these typical trajectories that are enriched in one of the two classes. And now I have a number of visualizations of showing like how you generate shapelet candidates. It's maybe not so important to go through every detail here now in the following, but I'll just highlight the most important aspects. So the shapelet in fact can start anywhere in the time series. We don't know whether the patients were admitted to the ICU at the same point of disease progression. So these time series are shifted, are asynchronous anyway. We don't know whether the disease, or the syndrome in that case, develops at the same speed. So we don't make any requirements on the starting point of a shapelet. It can start anywhere in the sequence. And now if we have two shapelets from two patients, then we overlay them and then compute a distance. So that can be the Euclidean distance as shown here. So here the squared Euclidean distance, or like a more time series specific distance such as dynamic time warping. So we quantify the similarity between two shapelets using a vectorial distance or some time series specific distance, and then if the distance is less than theta, as I mentioned before, then we say that this shapelet has occurred in this particular patient. But in and this is the main purpose of all of these visualizations, the number of candidate shapelets is of course enormous. Every subsequence, every time interval from every single patient is a candidate shapelet that could be statistically significantly enriched in one of the two classes. So we can represent each shapelet candidate in form of a contingency table where the rows here indicate the two classes. So y 1, those that develop sepsis, y 0, those that do not develop sepsis. i 1 means the shapelet of interest S occurs in these patients. i 0 means it does not occur in these in these patients. And now we would like to find those shapelets S where, according to a statistical test on these contingency tables, for example, according to Fisher's exact test, there's a statistically significant association of the occurrence of that shapelet and the future onset of sepsis.. But, and my group has worked on that for, for a number of years, there's now an enormous multiple testing problem that you have to solve. I mentioned every subsequence in every time series is a candidate shapelet for enrichment. So we are dealing with an enormous number of candidates and thereby hypotheses that we are testing.. And all of this, remember, also depends on a threshold theta that we have set. So we even have a more complicated multiple testing problem than in other scenarios because we also have to find the optimal value of this similarity threshold theta. And it can take, in a database of n time series, it can take n values. As well. So we have an— so the number of hypotheses grows in every dimension extremely. How can we deal with that, with this astronomical number of candidate shapelets and hypotheses that we are testing? Here it's stated that for 100 time series of length 100, which would be a very small and a very short data set in an ICU, we already have 10^8 candidate shapelets because the number of shapelets grows linear in the number of time series, quadratic in the length of the time series, m², because it can be of any, any length. So we can have any start and endpoint within the time series. And it's linear in the number of possible similarity thresholds, theta. There are n 1 possible thresholds. So the total number of tests is quadratic in the number of time series and in the length of these time series. So how do we deal with these, with this enormous search space? Well, we use a technique that we have now used in a number of studies, also in genetics over the last few years, namely the so-called improved multiple testing correction by Teron. It's a form of Bonferroni correction that excludes certain tests. It excludes tests of hypotheses that under no circumstances can become significant. And I'll not go through every mathematical detail here, but I'll just describe the main idea. So there is there is a frequency that is associated with a shapelet in a dataset. That's the number of patients that carry a particular shapelet. And Teron's argument for improving multiple testing correction is the following. I have a plot for that, I think. Yeah, this one is the following. If a shapelet is very rare in a given dataset— here in this plot, the x-axis is the frequency of a shapelet in a dataset from 0 patients to all n patients. So if the shapelet is very rare, then the minimum p-value that this shapelet can achieve in this set of patients is not very low. So in Fisher's exact test, if you think about it, and in related tests, if you fix the frequency of an observation, there's a minimum p-value that this observation can achieve. Namely if all instances of the shapelet occur in the cases or if all of them occur in the controls. And if the— so, and you can compute this minimum p-value, and this minimum p-value is quite large for shapelets that are very rare in a data set, and it's very large here for shapelets that are very common in a data set. The intuition behind this is if hardly anyone has a shapelet, it cannot be so strongly associated with developing sepsis. If everyone has that shapelet in the data set, it can also not be that strongly associated with having sepsis. And now when you test very many shapelets, as we do in this time series mining application, it turns out, and that's Teron's main argument, that the minimum p-value a shapelet of a certain frequency can achieve is larger than the Bonferroni-corrected significance threshold. So in other words, this shapelet is so rare or so common that it can under no circumstances reach statistical significance because the minimum p-value it can achieve is still larger than this, than the multiple testing corrected significance threshold. That's the idea. So the key idea is we ignore all rare and all very common shapelets. And now there are a lot of algorithmic details of that. So the number of shapelets and hypotheses that we correct for, of course, is interdependent on the number of shapelets that have a minimum p-value that could become that is below the significance threshold. These are the so-called testable shapelets, the shapelets that can reach significance hypothetically. The untestable shapelets are those that cannot reach significance. But there's, as I said, there's an interdependence between the significance threshold and a testable and the definition of a testable shapelet, because it is testable at a certain significance threshold. So you have to optimize both of these at the same time: the number of shapelets that you correct for, the number of testable shapelets you correct for, and the corresponding significance threshold. And we invested a lot of time into coming up with algorithms for doing that, for finding the optimal number of testable shapelets to account for. And the key benefit is that the number of testable shapelets will be much, much larger— it will be much, much smaller than the number of all candidate shapelets, and therefore we will very much improve the statistical power of our approach. I have one plot here where I can demonstrate this with an example from graph mining. In fact, so here on the x-axis we have the number of features. In our context, this would be the number of shapelets that we are considering. So in this example here, it's substructures of molecules, and we here quantify how many nodes there are in each of these substructures. And as we let the maximum size of these subgraphs grow, the number of tests, the correction factor here on the y-axis, goes to more than 10^9, 10^10 if we allow for subgraphs with 15 nodes. In our shapelet example, that could be like a time window with 15 observations. So the number of candidates we have to consider goes into the billions quickly, whereas the number of testable subgraphs, or in our case testable shapelets, remains around 10^4. That's a very interesting and very important empirical observation we made several years ago, that this difference between the testable shapelets testable patterns would be the more general term, between the testable patterns and the raw number of candidate patterns is so enormous. It can be like order 10^5 or 10^6 as in this example. So if we only account for the testable shapelets and the testable patterns, we much improve our statistical power and our ability to detect, in our scenario, shapelets that are associated with the endpoint of sepsis. And even on these old graph mining, chemoinformatics datasets, we suddenly found with this technique subgraphs that are associated with a particular molecular function. Good, so I'll keep this part very short. I just point to our work. In this domain. So we worked on how to efficiently find the optimal number k of testable hypotheses. We developed a scheme how to account for the fact that there's a subset-superset relationship between shapelets. So some shapelets, some time windows are contained in another time window. So there's a subset-superset relationship between them. How can we account for that in the testing? We do so by Westphal-Young permutations. How can we account for categorical covariates such as age or gender for the patients? We came up with a solution for that based on the Cochran-Mantle-Hensel test. We developed a concept of testable hypotheses in this Cochran-Mantle-Hensel test. And can we develop in genetics or in the intensive care unit new association mapping approaches based on Teron's trick. So for a number of years, we worked on this in the context of genome-wide association studies, and in 2018, we started using this technique also on time series. What I described before was our work in this direction. This is the paper book at ASMB 2018. So we here— I'll cut this also rather short— we came up with an efficient way of determining the number of testable shapelets in such a shapelet enrichment problem setting. Good. And then we applied this to sepsis patients, trying to identify shapelets that precede an onset of sepsis. Now, I could give a full talk about the question how to define the onset of sepsis. It's not very clear, or not unique at least, how this can be done. How it's— so like the realm of possibilities that is currently being used in literature is the following. So according to the Sepsis-3 definition, there must be some suspicion of infection by the doctor. That usually means that the doctor gives antibiotics to the patient, or takes a body fluid from the patient. And roughly at the same time, there must be an increase in an organ failure assessment score, in a score that reflects organ failure, the so-called SOFA score. So both this suspicion of infection and the increase in the SOFA score define in their co-occurrence the onset of sepsis. So that's the endpoint that we defined. And in fact, we were the first ones on these big public datasets to precisely define the onset of sepsis in these datasets according to the international definition of sepsis. Good. And then to cut a long story short, we discovered, in fact, shapelets that were associated with developing sepsis. So in heart rate, we had this variability in the heart rate. So one time step here is 30 minutes. So over a longer time horizon, heart rate was going up and down in the sepsis patients. So that could be one potential biomarker. In the respiratory rate, we saw a drop into an abnormal regime followed by a sharp increase. This drop into an abnormal regime has in fact been reported in the medical literature as a phenomenon, as an observation associated with sepsis. In the systolic blood pressure, we discovered a characteristic spike in those patients developing sepsis. Of course, this is like only the proof of principle so far. We now have to take these findings to other datasets and see whether we can reproduce them there, whether we can validate them there. We are actively working on that even at a national level. I'll come to that in a minute. We have developed software for this task of significant shapelet mining. Even if you mine significant shapelets, there's still a large number of them, so you get very many shapelets that are quite similar to each other. So this year we had a paper at ECCB, Gumsch et al., that clusters these shapelets and picks representative shapelets in this large space without destroying the validity of Teron's trick. So you can use it yourself, I'm happy to share the slides. Here I summarize the findings that we have made in terms of significant pattern mining, but There's no need to go through it slowly now. I've mentioned the most important things we did already. So, so this concludes the first long part about association mapping in time series. I'll now quickly summarize in a few minutes the second part, namely predicting sepsis through time series classification. That's the more classic approach. That's the approach where you put the entire time series into a, into a machine learning algorithm, and then the machine learning algorithm outputs a prediction of yes, there will be sepsis in the future, or no, there will not be sepsis in the future. So we reviewed carefully the work that has already happened in this field, and we noticed a number of concerning facts. So one is that even on the same dataset, a very commonly used dataset for this task is Mimic-3, even on Mimic-3, there the number of sepsis cases that authors report can greatly vary. On the same dataset. That is very worrying. So even on that, there is no consensus. So we see here that even when Calvert et al., Kam et al. used the same dataset, MMIC-II, the prevalence of sepsis within their study can differ by almost 5%. And it varies greatly among these different studies here as well. This is due to— mainly due to the fact that the endpoint is defined differently. So some studies use this newer Sepsis-3 definition. Some use Sepsis-2 or a variant thereof. Often they use variants if they don't have access to all variables that are necessary to compute the official full definition. Of sepsis, or they do a proxy of that, that they use an ICD-9 code and look at the SIRS score. So there are very many ways of defining sepsis, and that leads to a different prevalence of sepsis in all of these datasets. And some of them, of these approaches, in fact, are now being tested in randomized clinical clinical trials. So that's, that's an important point to make. So they are being brought to the clinic and they check whether patient mortality reduces or length of stay reduces if the doctors have access to the predictions of these tools. But to cut a long story short, it's extremely hard to compare the performance of these different approaches because they differ in every aspect in the definition of the endpoint, in the exact time point associated with the outbreak of sepsis, and even how they evaluate may differ in different studies here. So this is a summary of the different results that were obtained with these methods. So how The problem is usually approached is that you try to find out whether X hours before sepsis onset, the algorithm can predict that there will be a septic event in the future. So the X-axis here is the hours before sepsis onset. And the Y-axis is the area under the ROC curve or the area under the precision-recall curve. The further up we are, the better, and the further up we are at the left end of the y-axis, the earlier we can make good predictions. And I've mentioned that comparing different algorithms that address this problem is extremely hard for a number of reasons, that the label definitions are heterogeneous, that label extraction is heterogeneous, that the studies don't provide code for extracting the labels or even code for running their predictions. There's a danger of circularity that you use the same variables or same observations for defining sepsis as well as to predict sepsis. There's a difficulty in how to define control patients, patients that do not develop sepsis, which time window in their health record do you use as a control? And another problem is that only few studies report precision and recall despite a considerable class imbalance between sepsis cases and non-sepsis cases. We did a systematic review on that, Moore et al., which is on medRxiv now, so you are welcome to to read all the details about the difficulties in comparing sepsis prediction in the literature. We developed our own approach for this task based on a multitask Gaussian process and a temporal convolutional network. So the multitask Gaussian process for imputing missing values in the time series, which is a very common problem, and the temporal convolutional network for making The predictions we compared to classic tools from data mining, dynamic time warping with a k-nearest neighbor classifier. We again defined sepsis on the mimic3 dataset. We optimized the parameters of our deep learning approach with the multiple multitask Gaussian process integration imputation of missing values in an end-to-end fashion. And we then obtained the following results, and that's worth to look at that for a minute. So interestingly, and that's also a take-home message for me, the area under precision recall curve 7 hours before the onset of sepsis is highest for the classic approach from data mining. Our new approach outperforms, in dark red here, the MGP TCN, it does outperform like a simple temporal convolutional network and this published multitask Gaussian process with a recurrent neural network that is in a clinical trial at the moment. So these are the green and the blue lines. So we are better than those. That means in terms of the neural network architecture, there is certainly room for improvement.. But 7 hours before sepsis onset, in fact, the baseline is best in terms of area under the precision recall curve. So it's one take-home message. Also compared to the classic methods, don't take it for granted that the deep learning approaches will always outperform everything. And we see the same picture here in area under the ROC curve, also with a time horizon of 7. Hours. So here at the beginning, the dynamic time warping with k-nearest neighbor is best. Good. I mentioned all of these points here, the lessons that we have learned. Now, what's coming next? Well, I mentioned that we are doing this at the national level. So together with Adrian Egli from the University Hospital Basel, I'm coordinating a Swiss effort to harmonize Swiss ICU data from all Swiss university hospitals and to do joint sepsis prediction on this harmonized data. We have been working on this for more than 2 years now. Now the data is flowing after solving a lot of legal and organizational issues, and now we are collecting in the order of 800 to 1,000 electronic health records per month. So our dataset is now rapidly growing. And we plan to do sepsis prediction using routine electronic health record data on this dataset. And then in a second phase, we also plan to collect multiomics data on these patients and to predict sepsis based on these multiomics, on this multiomics view. We did a similar study with colleagues from the computer science department at ETH around Gunnar Retsch and with colleagues from the Inselspital Bern around Tobias Mörtz for predicting circulatory failure in the intensive care unit. And this we could do with a high recall and a very acceptable false positive rate. And this work appeared in Nature Medicine earlier this year. Good. So to finally conclude, I'm convinced that automation, biomarker discovery, and biomedical data management will remain key research topics. The data will continue to grow in 3 dimensions. So in the number of patients, in the length of the observations, and in the depth of the observation from the molecular level to the higher phenotypic level. Big question for for us machine learners and for the medical researchers is how to mine, how to handle and use this data. For which purpose do we want to use it and how can we achieve that? I talked about intensive care units today, but many branches of the life sciences face very similar or analogous problems. We also work a lot with the synthetic biologists at our department. They have, in fact, very similar questions, at least on an abstract mathematical level. So I'm convinced that there are plenty of opportunities for machine learning in precision medicine, that we won't run out of exciting problems anytime soon.

I thank you again for inviting me, and I'm happy to take your questions now. And I thank my group and our sponsors. Thank you very much, Carsten, for this comprehensive talk. And now we have the opportunity to start the interactive discussion round, and For this, you can all turn on your screen if you have a question, but first of all, I would hand over to Lena to start with the first question, if you want.

Okay, thanks, Doreen. I think we should keep only a few minutes because I think people have to leave. They normally leave at 12 o'clock. But yeah, many thanks, Carsten, for this. Very exciting. You know, we always get asked by our clinicians how much training data do they need to provide or annotate?

And I was wondering if you have looked into this aspect of how much do you actually need to get similar results? So that's a very interesting question because— so when we apply Terence's trick, the perspective is a bit different. We start from a given dataset, so all the algorithmic magic we do there is making one strong assumption, which is that we have a dataset of finite size, and our ability to search for— to search these enormous spaces of candidate shapelets depends in fact on this assumption that we have a dataset of finite size. So it's the reverse. We start from a fixed dataset size, and then if it is manageable enough, we can compute, we can search very complicated search spaces efficiently. Now, in the sepsis scenario, it's very hard for us also to answer this question because we don't know the effect size of this, of the biomarkers that we will find. So it's not a question I can easily answer or fully answer. At the moment. In retrospect, I might be able to answer it. I talked to very famous statisticians at ETH with exactly this question, and they also said, like, coming up with a number here, that would be kind of black magic to know in advance, like, how many sepsis patients you really need for making this predictor work. You can make strong assumptions on the effect size, and maybe then you can compute it.

But whether these assumptions on the effect size are realistic, that's a totally different issue. Yeah, I mean, maybe empirically you could even find something, right? So if you, for example, your shapelet method, if you just cut your data in half and would basically see whether you make the same findings then.

The problem is, the problem is that the shapelet method— excuse me— the problem is that the shapelet method is conditioned on the size of the data set. So I totally change everything if I compare it on data sets of different size. So this conditioning on the data set size is the strong assumption we make. And yeah, one could get some weird effects if one tried out different data set sizes here. But in the sepsis setting, so in the predicting sepsis setting with national scale data, of course, a fundamental question, the one you asked, like how big should these datasets be?

Right now, our attitude is as large as possible.

Doreen, there's also questions in the chat that have been asked a while ago. Yes, I can see them. In fact, maybe I could just go through them in the interest of time. Yes, that would be perfect. There's a question about the p-values and the the class sizes, the class ratio between cases and controls. Teron is not sensitive to that because Teron operates with the minimum p-value, the Teron strict, and the minimum p-value is in fact irrespective of the class assignment of the shapelets. So whatever the class assignment there is this— this should, it should be robust to that. Yeah. And regarding the representative associated shapelets, are these frequent enough, for example, to be embedded in advanced heart rate monitors so that these automatically get flagged? So essentially turning this into a real-time analysis. I— that's our dream, but we are not there yet. So right now we are doing it retrospectively. On data that has already been collected. There will be some interesting effects if you do this, what we do here online, which is that shapelets could become significant or lose significance over time. So we haven't studied this problem yet. Our vision of using this in practice would be more that you discover interesting shapelets retrospectively on a given dataset and then in a new patient, you check whether this shapelet occurs in that patient so that you treat the set of relevant shapelets as fixed when you apply it to a new patient. So that would be, in my opinion, the easier way of applying this in practice. Now, how does your work to extend Jerome's multiple hypothesis testing procedure relate to independent hypothesis weighting? Is it helpful to go from discrete cutoff value to a continuous weighting? I think these connections still need further exploring. I think they are not directly related, but they, when combined, you may still get a more powerful technique. So Teron-Strick is very much based on the finiteness of the dataset, on the discrete the discreteness of test statistics and thereby rules out that certain candidates can reach statistical significance. And I'm sure they're— and I'm very interested in that. I'm sure there might be clever combinations with other schemes of multiple testing correction. Of course, like a common question is, can you do this with false discovery rate control? You can mathematically, but you cannot compute it efficiently so far. That's the answer to that. So that's why we control the family-wise error rate with Bonferroni correction so far with Terhoen's trick. It could be extended to false discovery rate control, but so far we don't know how to do this without losing efficiency. Good. Any further questions?

If not, then thank you very much for the interest. And the questions and the invitation. Yes, thank you for being here. Okay, we are at the end of our seminar. Thank you very much also to the audience. So see you in 2 weeks.

Bye!