Scientist Stories: Ulf Leser, Precision Oncology and Data Analysis

So good morning everyone and welcome to the NCT Data Science Seminar and a Happy New Year. This year also starts with an innovation: the Data Science Seminar series united with Heidar and today we will present the first Heidar lecture. It's my pleasure to welcome Professor Ulf Leser, our speaker today, Professor Maja Hein sharing this event and Victoria Schwarze as representative for Heidar. After the talk, as always, There will be the section for questions and answers. So please use the chat function and raise your hand. I hand now over to Victoria Schwarzer for a short presentation of Heidum.

Thank you, Doreen. We will start with a short presentation of our Helmholtz Information and Data Science Academy, which is of course part of the Helmholtz Association, which you probably, most of you know, but probably we have also some visitors from outside. So Helmholtz is Germany's largest scientific organization with 18 research centers. And as part of that, HeDa is connecting and serving as a roof of 6 data science research schools and is with that Germany's largest postgraduate training network. Here you can see all of our 6 Helmholtz Information and Data Science schools throughout Germany, and we have the honor to be here today with the HITS4Health. The, the activities offer high variety. We have training and networking events. As you can see here, we have exchange and visiting programs. And of course, we promote and provide financial support to all Helmholtz institutions. And furthermore, we have a very interesting data science job site on our website where you can see a lot of different data science job, jobs throughout the association. And if you want to know more about Heeda and our activities, we would be very happy if you follow us on Twitter, LinkedIn, or just see that you get our Heeda newsletter with our newest activities. And that was our very short presentation, and I'm very happy that we can hear today more of, of Lisa. And with that said, I'm handing handing over to Klaus Meyerheim.

Thank you, and good morning also from my side. It's my honor to host this first joint lecture between HeDa and HITS4Health. I am speaker of HITS4Health, and it's really my pleasure to welcome today Ulf Leser. He is— yes, and also Ralf is here. Hi, Ralf. He's also speaker from HITS4Health. My pleasure to host Ulf Leser today, who is a full professor for computer science at Humboldt University in Berlin. And he's also principal investigator in two interdisciplinary graduate schools, the Computational Methods for Precision Oncology, CompCancer, and the Helmholtz Einstein International Berlin Research School on Data Science. Ulf Leser started his career with a PhD in data integration. Also in Berlin, and he has gained 3 years of experience in software industry before he became a professor for knowledge management and bioinformatics. And yeah, his main research interests range from what I can tell from the full range from algorithm development also to up to interdisciplinary applications. He has a strong focus on cancer research including topics like semantic data integration, text mining, statistical bioinformatics, and also large-scale scientific data analysis. So I'm really looking forward to your talk, and the stage is yours.

Thank you very much. Thank you very much for the introduction, and thank you very much for the invitation. I'm quite happy to give this talk, as you If you didn't hear it between the lines, so this second international interdisciplinary, interdisciplinary graduate school that Klaus just mentioned also is part of HeDa. So the hybrid school, Helmholtz, actually always forget the full name of the abbreviation. Helmholtz-Einstein Interdisciplinary School for Data Science is, I think it was the first of these Helmholtz graduate schools and it then has become part of FIDA and we are now at the moment negotiation for an extension. So I think I know the setup of HITS4Health quite well because of this. Yeah, so my presentation today will be on data science for molecular tumor boards, for subtyping molecular tumor boards. It's mostly results of a BMBF project that was called PREDICT, predict for Comprehensive Data Integration for Precision Oncology. We also had a sister project that was called Pearsons, which had a kind of similar range. And today I'm kind of presenting the results of our contributions to do the— to these two projects. If you have urgent questions, interrupt me at any time. I will also have time after the presentation, so I will not run away. Either wait with your questions or just just interrupt me. Yeah, we are in the field of— do you see my laser pointer? Yes. So we are— ah, you stopped my screen sharing. Of course, you have to share again. So do you now see my screen? Yes. And the laser pointer? Yes. Okay. But I— aha. Okay. I have to use the keyboard. I don't know why. So the field we are— I will be talking about is precision oncology. And just in a very short nutshell, in a very small nutshell, so a very brief introduction into the general idea of precision oncology. So the main idea is that the plan is to determine therapies for cancer patients depending on the molecular characterization of the individual patients. So this is not quite state of the art today. Mostly therapies today are based on large studies, on large clinical trials, and the specific individual makeup of the participants of these trials are typically not really taken into account. But the hope is that now that we are able to do these large screens relatively cheap of molecular characterization, so transcriptome, genome, metabolome, and so on, the hope is that we can find the specific information that is causal or important for the specific cancer in the specific patient and then find a targeted therapy directly targeting this individual characterization. So a very important part of this molecular characterization is the genome. There are also other areas. Today we only talk about the genome. And in this genome, an important part are the mutations or the variations. There are also other parts, but today we focus on variations. Cancers always have different variants, so it can be between a few dozen and a few hundred. That depends on the cancer type. And there is a growing number of knowledge that different variants or different combinations of variants lead to different prognosis and more and more to different treatment. So it's a classical task to, to have a certain cancer type to sequence many patients and then trying to find subgroups of these patients that, for instance, have a longer overall survival or have a shorter overall survival. Now, precision oncology is about these targeted therapies. So here the idea is that we find certain variants and these variants destroy the functioning of a certain protein. And the question is, can we somehow block this protein and block this functioning of the protein that is only going wrong because of this variant? So if we know the variant, we can then say, okay, patients with this variant in this cancer type, we should use the following drug. So the targeted therapies are, in short, drugs whose applicability depends on the presence or absence of certain variants. And this is becoming more and more popular. So there was a recent publication, then they claimed that one-third of all new anti-cancer drugs in 2019 at the FDA, at the Federal Drug Administration, so the, the institution in the U.S. that surveys the admission of drugs, so that one-third of all new anti-cancer drugs actually were targeted therapies, and that 90% of all current late-stage anti-cancer drug developments are also targeted therapies. So that means that the medicines that are now becoming, that are coming to the market and new, are mostly targeted therapies. Now, the question is, how can we decide whether a given variant is a promising drug for targeted therapy, of course we need a sufficiently large group of patients where we find this variant and where we can test our drugs. But that is often very difficult, especially for rare cases. So rare cancers, cancers that don't appear very often, that don't happen all too often, because for these rare cases it's very difficult to find enough patients. To have a, to have a really statistically valid statement about the, about the effect of a certain drug. For these rare cases, this leads to the case that the doctors, the practitioners, the oncologists that are trying to find a drug, that are trying to find a therapy, often have no clinical guidelines available. Because the clinical guidelines are based on large clinical trials and for very rare cases, you don't have these clinical trials. So the question now is, we have information about drugs that may target certain variants. These have probably been tested in large cohorts, but only for relatively popular or relatively frequent cancers. But now we have also cancers where the decision is much more difficult and where we now have to— where the oncologists have to find a decision on what they do. With these patients. And these decisions today very often are taken in so-called molecular tumor boards. So tumor boards are there since ever. A tumor board is a, is an interdisciplinary union of doctors from different disciplines trying to give a comprehensive survey on the status of a certain patient. And this is now more and more turning into molecular tumor boards as more and more molecular information is actually becoming part of the game and more and more patients are discussed where we know about the molecular makeup. So the decisions that these MTBs, Molecular Tumor Boards, are taking are more and more based on molecular data, especially genomic variants. As I said, today it's only about genomic variants. So the input, let's say, of a Molecular Tumor Board is a patient. It's a difficult case, so it's outside the typical clinical guidelines. We have a molecular characterization which here boils down to a list of variants. So we know It's pancreatic cancer, and we know that the certain 60 genes have the following mutations. And the question that now has to be answered in the Molecular Tumor Board is what is a good therapy for this patient. So the entire decision process is long and very, very complex. So it starts from how did we take the sample. Today, we could do single-cell sequencing or did we bulk sequencing. How was the sample handled? How was the sequencing done? How did we actually find the variants? Did we— can we compare our variant calling to our variant calls in the past year or in the last decade or to other hospitals because we all have changing pipelines that lead to variations in the actual results? That creates this list of variants. Now that we want to take a decision based on this list of variants, we all, we of course also need knowledge about these variants. So we need to know, okay, this variant, what is known about it? Is there some targeted therapy? Is there a targeted therapy against this variant in another cancer type? Are there some studies where, for instance, in mice, it has been shown that a certain drug might be effective against this variant? Do we have information from, from in silico predictions? Do we have information from cell lines and so on and so forth? When we have all this annotation, we have to take a decision. So the oncologists have to take a decision, and what is then also very important is that we can, that in the future at least, we should be able to have a reproducible and transparent reporting. So the decision has to be taken on rational grounds. So we have to lay down a document where we can say at the time when we took this decision, we had the following information, and based on this information, we took the following decision. Yeah, so this variant annotation, so the information about the variants that we have to lay down in reporting is then the basis for the actual decision support. And of course, taking a decision, doctors have to take much more into account. So they have to look at the entire patient's health or the history of the patient, family history, medical data, and so on and so forth. And then they eventually say, okay, we will use the following therapy or not. In this talk today, we will only focus on the blue stuff here. So on the variant annotation, I will actually not really talk about decision support, but about systems that provide the information that is necessary for an informed decision support. Now I say that in the future, we should be at the stage where we have reproducible and transparent reporting. So I actually kind of claim that we don't have this today. So what is the state of the art today? And this is the result of a paper that we did in this project that appeared in 2018. It's now a little bit old, but I don't think that this situation has changed much. And what we did here is we set up a synthetic and explorative trial. So our question was, when we have a patient or a group of patients, and we know the variants in these patients, will different molecular tumor boards give the same recommendation? So if they would all give the same recommendation, then we would have a very systematic process. If they have the same information about the patient, if they know these are the variants, then in some sense, there should be a best therapy. And then the best therapy should be always the same. But this is not at all the case. So here, Damian was the first author, he made up 4 realistic patients, but synthetically. So we looked at real medical cases, did a certain classification. But at the end, we had a description of the patients and he had a list of variants in all of these patients. And he contacted 28 molecular tumor boards. It was Already a little bit disappointing that only 5 of these answered. So the other 23 were not really willing to— I don't know, let's not speculate about their motivation. But already between these 5 molecular tumor boards, the therapy recommendations were completely diverging. I think there was one patient where 2 of them gave the same recommendation. For the other 3, 5 MTBs, 5 recommendations. So that was a sign that the current state of the art in molecular tumor boards is definitely not systematic. And that different molecular tumor boards apparently have access to very different kind of experiences or very different kind of knowledge bases or very different kind of knowledge, and that this leads them to very different recommendations. We don't know what the best recommendation would be here. So that's not the claim we did. We cannot say that anybody did a wrong decision. We can just say that they take very different decisions. And with this small trial, we also had a survey. And one important result of the survey was that, well, we essentially asked the MTBs, what is your decision based upon? And the most important information by far, the most important source of information was PubMed. So the doctors, the oncologists, when they see these variants, they know about the patients, what do they do? They sit down to PubMed and start browsing the literature. They start searching for new trials, and they try to find the information that gives them the decisive hint on how to proceed. Okay, so now predict this project that I'm reporting upon is about knowledge management or knowledge infrastructures for molecular tumor boards. And we now learned that the essential knowledge is mostly in the literature, in scientific articles and trial reports, maybe in meeting abstracts. So ESCO and ESMO are very important sources. Even at the abstract level, not, not even far before the full paper actually is published somewhere. This literature knowledge is partly curated in precision oncology knowledge bases. So there are curated databases like CIViT or OncoKB, where professional curators try to catch up with the literature and then get the information and store it in a structured format. Which essentially boils down to something like in this type of cancer with this variant, there was a paper saying that the following therapy had the following effect. So we were now thinking about how can we support this process of taking decision, how can we provide knowledge from molecular tumor boards. And we identified 3 different ways how we could do this. And we then worked in all these 3 directions. And I will talk about them now one after the other. And the first one is that we said, okay, if people are using PubMed, can we build a better PubMed? So can we improve information retrieval through a search engine? And for this, we build a tool, the VIST, the Variant Information Search tool, and I will introduce you shortly to this tool on the next slide. The second topic will then be, if there are these precision oncology knowledge bases, and people are using it, The problem is that there are quite a few of them. So can we build an integrated variant database? Can we build an integrated databases where we curate different, where we integrate different curated databases into a single system as a one-stop shop where oncologists then get all the information that originally started different databases? So we want to save a lot of time and also report on conflicting information there. And we built a system, it's called IVIS, an Integrated Variant Information System, that would be the second part. And the third part of the talk is, well, if most of the information is in the literature, and reading literature takes a lot of time, which is one of the reasons that these positional cody genomics bases are all very, very incomplete, so that there's always a lot of information lacking, it is not stored in a structured format. So the third line of research would then be, can we do— can we automatically extract knowledge from the relevant documents? And that's a field of information extraction or text mining. And this will be the third part of my presentation. Okay, let's talk about the first, building better search engines. Now, what was our main idea here is when you think of PubMed, probably all of you know PubMed, then you essentially search by keyword. So there is a certain additional features that are part of PubMed, so you can search for structured information like only in this journal or only in this year or only by this author. You can have a certain type of ranking. So the standard ranking is by date. You can also have a date by— a ranking by relevance. But the relevance that is computed for an abstract given your query is a standard information retrieval relevance. So essentially they embed the documents and they embed the query in a high-dimensional space, in a vector space model, and then they compute a relevance based on the overlap, essentially on the overlap of terms in the query and terms in the documents. So the documents here are mostly abstracts. And when we talked to our clinical partners in our project, that their main problem with this type of ranking was that the papers that they see very often are clinically very, very irrelevant. So the first 20, 30, 50, 100 matches or whatever that they find when they search with variants or with genes are about cell lines, are about mice, are about preclinical study. All this is very interesting from a basic research point of view, but we are now talking about molecular tumor bites, tumor boards, and they don't want to do basic research anymore, they want to take a therapy decision. So their main problem is the lack of clinical relevance in many of the hits that they get with PubMed. So that was the central issue, how can we filter or rank publications by their clinical relevance? So we did not improve the matching of a variant to a document, but we wanted to have all the papers that match a certain query, we want to re-rank them based on the clinical relevance. And how did we do this in VisST? Well, we essentially did it by text classification. So we collected a set of documents where we were absolutely sure that these are clinically relevant, that they contain clinically relevant information. We did this simply by taking papers that are cited in OncoKB or in CIViC. And these are the databases about clinically relevant information between variants and cancer. So we said whatever paper is cited there should be clinically relevant. And based on these set of documents, we then trained a classifier. So classical text classification system that given a document can compute a score and the score is the higher, the higher the probability is that the document itself is clinically relevant. The nice feature of this approach is that we can actually pre-compute this score score because it's not dependent on the query. So we can, we essentially precomputed these clinically relevant scores for all abstracts in PubMed. And when we now see a query, we can access these scores and we don't have to do this on. So this is how the entire pipeline works. So let's start here in the middle, we have a full copy of PubMed, we first recognize all genes and all variants. This is named entity recognition. So we'll talk about this a little bit at the end. But essentially, we try to resolve homonyms and synonyms and abbreviations and really say, okay, this is a certain gene in this paper. And we have an idea of this gene, so a RefSeq ID or something like this. And this creates us an annotated version of PubMed. So where for every abstract in PubMed, we know which gene and which variants are contained. A second As an input, we have this classification model that is able to compute scores regarding clinical relevance. And this is trained on CIViC and on Oncocare B. This gives us a classification model and we can apply this classification model on all of PubMed. And this gives us a pre-ranked set of documents. So we now have a query-independent ranked set of documents where the top ranks are those that are most clinically relevant. Now we have our query. So here we have this, this pictogram of a, of a profile of variants. The user of the system has to do a certain prioritization so he can search the system with 1, 2, 3, 4 variants but not with 50. So he has to somehow select these are the most interesting ones. And we then match the set of variants, or we can also search by genes to the set of annotated PubMed document documents. This gives us a list of matching documents and these matching documents are then presented in the rank that we precomputed based on the clinical relevance. So what a user has to do here is two things. He actually only enters the variants that he wants to search for, and he then gets the result, the list of documents. Yeah, this is how it looks like in operation. So here we search with a certain mutation, as you can search as well search with the gene only if you don't know the mutation but only that a certain gene was affected. And then you can, again, you obtain a list of documents. Here you have a score. So this is the score that we computed regarding clinical relevance. You have certain filters. So you can search only here in certain days, or you can filter by journal. You can also click on these and then you get a view of every abstract where we highlight the genes, the mutations, and the drugs. So here you can see that CKRAS is recognized as a gene. We also do a highlighting of the most important parts of an abstract. So we do a key phrase extraction algorithm, we run a key phrase extraction algorithm here. You see that the display is not perfect. So here you have mutations. Somehow there was a blank inside which we didn't recognize. And what we also integrated is you can see it here as a second tab. So this is the display of a Medline abstract, but you can also look into clinical trials. So we downloaded clinical trials, also run our pipelines over clinical trials, and also recognize gene, chemicals, and mutations, and can now also give you a list of clinical trials. Yeah, building a search engine is nice, but of course you also want to know whether it's any better. Than the state of the art. And this is a very, very difficult undertaking. So whoever has worked in search or in search engine knows that evaluation is always very difficult. So what we try to do here is where we actually try to obtain a gold standard. And for this, we asked for experts. We chose 20 variants in discussion with these experts. For each of these variants, we chose 10 documents.. And then we asked every medical expert, could you please, given this variant, tell us whether this document is clinically relevant, somewhat relevant, not relevant, or wrong, because it doesn't mention this variant. So we had a 5-point Likert scale for these assessments. We altogether obtained 188 assessments. So it should have been 200, but not all experts delivered all assessments in time. Roughly 40% of the documents were overall assessed as highly relevant. Roughly 40% were assessed as, yes, this document talks about this variant, but it's not clinically relevant. And roughly 20% were simply wrong. Now, the problem is we had to reduce these 188 assessments because we had overall a very low inter-expert agreement. So this reflects what I told you about the low agreement between different molecular tumor boards. So these were 4 experts from the same molecular tumor boards. I think 2 were oncologists, one was a pathologist, and I— and the other one was a pathologist as well. And they simply don't agree. What clinical relevance actually means. So we have papers you see here at the bottom, the query is a variant. Now we have the PubMed IDs, these are the 10 documents. And then you have entries like this one here, where the expert 1 says, yes, this is very relevant, somewhat relevant, very relevant, not relevant. We have papers like this, all agree this is very, very relevant. And we have other papers where the scores range between 2 and 3, 2 and 4, 1 and 3. We also had papers where we had ranks between 1 and 4. And we somehow, when I say that 40% are highly relevant, we had to do an aggregation of this. And this aggregation essentially, we did it in 2 types. So a loose and a strict. In the strict, we only said something is relevant when they all agree that it's either relevant or somewhat relevant And in the loose aggregation, we had a more loose aggregation strategy where we actually allowed more disagreement between the experts. Yeah, and with this set of documents, we then did a comparison to other systems. Yeah, so we had to do this filtering. We had these cases that are where the agreement was so low that eventually we only could proceed with one of the one assessment. And then we did an evaluation. We actually compared WIST without the clinical relevance scoring, WIST with the clinical relevance scoring, and the usual PubMed search engine. And for this, we really use PubMed. Now, the evaluation, of course, again, is very tricky, because when you use PubMed, you don't only get these 10 results, these 10 documents for which we have an assessment. So we— what we did is we looked at the average ranks at the ranks, and then we averaged. We looked at the ranks that our documents obtained, and we compared the documents that were classified as clinically relevant, averaged the ranks, and compared this to the documents that were classified as not clinically relevant and averaged the ranks. And this gives us a score when we put this into relation where lower is better, because lower means that the clinically relevant documents were ranked higher. And here you can see that the red, which is our system on average, performs quite better. It's not always better. So here query 2 was a variant where actually PubMed ranks better than we do. But overall, the red ones are lower than the green ones. And there are more evaluations in the paper. We also published on the corpora. So you can have a look at this. Okay, the second line of research that we proceeded is that we said, okay, there is certain information that is available in structured format. So let's harvest this, because searching PubMed is always a little bit clumsy, let's say. Why is this a useful undertaking? Because there are quite a few of these knowledge bases. And this is simply the case because Molecular Tumor Boards are not a German not specific for Germany, but of course they emerged all over the world, and in some countries they emerged much earlier than they did in Germany. And so there were people discussing hundreds of patients over years and taking decisions, and at some stage they simply started to write down which decisions they took and why they took it. And this led to the creation of a, of a whole selection of what we call precision oncology knowledge bases. Which essentially contain information about variants of genes and drugs. And they're all associated by experts. And here, just to name a few of them, CIViC, OncoKB, Anderson Lab has one, the Precision Medicine Knowledge Base, DOCOM, the Jackson Lab Knowledge Base, MolecularMatch, myCancerGenome, and so on and so forth. You already see that quite a few of them are now commercial. So MolecularMatch is a company you cannot access the knowledge base that they use. My Cancer Genome used to be open, but it's now also only commercially available. That is a real danger for this second line of research, that more and more of these precision oncology knowledge bases become commercial, and that we can then by license not integrate them anymore. So this is an issue that we are facing more and more. Now the problem— so it's very nice that there are so many precision oncology knowledge bases. The problem is that there are so many. So for a single oncologist, biologist, he or she now faces the problem that you can spend quite a lot of time in surfing different databases. And we wanted to build a single integrated database, and that's what we did. That's also useful. So we built a database where we integrated 6 of these knowledge bases. And here you see, I mean, one could speculate that maybe there is only a a small set of oncogenes, and maybe all these precision oncology knowledge bases therefore talk about the same genes or maybe the same variants. And then of course it suffices to ask only one. But this is not the case at all. So here we did a very simple overlap analysis. We looked into biomarker targets, DG-IDB, CIViC, and OncoKB, and you see here this Venn diagram, and you see that each of these knowledge bases has differently large set of genes that is only discussed within this gene. So this here is at the gene level, not at the variant level. And that there's only a quite small set of genes that are actually present in all these knowledge bases, and that the overall number of genes mentioned in any of these databases is quite large. So it's more than 4,000 genes. This is mostly due to DGIDB, which itself is a very large database. So it absolutely makes sense to look into more than one of these databases because you will always find more and more and more information. Yeah, then we built this integrated database. Sorry. So the most important information is here to the right. We integrated CIViC, OncoKB, Dockum, PMKB, CGT, TARGET, and BiomarkersDB. We also integrated a lot of annotations. So we integrated information from DrugBank regarding the drugs that we are actually storing. We integrated information from more disease-oriented but not cancer-specific databases like ClinVar, DGIdb, and the Disease Ontology simply for as a normalization target for naming diseases, in our case cancer types. We integrate information about variants from dbSNP, COSMIC, and dbNSFP. We integrate information from clinical trials, and we also have some frequency information from these large-scale scans, so Ensembl, ExAC, or the 1000 Genome Project. Yeah, our core entities are variants, genes, drugs, and cancer types. So a variant is in a gene, it is targeted by a drug, a drug can also target a gene. And we have information about the role of this variant, or this gene, or this drug in a certain cancer type. And here you see where we get this information from. So regarding genes, we have quite a lot of different information sources providing different aspects. Regarding variants, it's already less. Regarding drugs, it's less, and regarding cancer types, it's also less. Then a very difficult part in information integration is always if we find a certain drug in one database, and we find another drug in another database, we actually, in an integrated database, you want to have this drug being represented only once if it is the same drug. So you have to map all the names, all the identifiers of your entities into a common namespace. So for variants, we here chose HGVS, so the official nomenclature for variants. For genes, we chose HGNC. For cancer, we chose the Disease Ontology. And for drugs, we chose Campbell. And this normalization, so the mapping of a name or ID that you find in the database into a common namespace, actually is a very, very difficult problem, and it's often underestimated. So when you as a user use these databases, you do it yourself. But if you do this systematically, programmatically, if you build an integrated database, you have to write programs for doing this. And even for something presumably simple like cancer types, this is actually very difficult. So there are let's say at least 100 different cancer types. And for each of these cancer types, you have different names and different ontologies. And the ontologies themselves are not really clean. So if you look into something like cancer, colon cancer, and you would use the MeSH, for instance, the Medical Subject Headings that are used for annotating PubMed abstracts, then you find 20 or 30 different instances of colon cancer in different spelling variations, in different branches of the tree at different levels of of abstractions. And it's quite difficult to map then all these into one only ID for making clear that we now have all the information that we want to have about colon cancer, for instance. So here we chose the disease ontology because it's a relatively clean ontology. But now you have cancer types are typically not represented by IDs, but really spelled out names. Now you need a mapping from these names to the instances, to the occurrences of these names in the disease ontology. That is already difficult for genes, because people don't use standards. It's very difficult for drugs, because there are so many ways how you can name a drug. It's manageable, let's say, for cancer types, because there are not so many. And it's kind of impossible for variants. So variants, for variants, we didn't really try, because there are so many ways how you can name a variant and these different names are used in publications in the different databases, and it's extremely difficult to map this into a common representation. At the moment, our list of variants is not redundant-free. So it is not free of redundance. So, so we will have, we will have many variants that appear multiple in our database, because we just didn't yet find a way how we could normalize this. Yeah, just as an example here for drugs. So that already is surprisingly difficult because you can name a drug by the chemical substances that are the actual drug. Then you have the products. The products can have different names in different countries. Then you very often also, well, doctors very often mention drug classes and not individual drugs. And you all have to somehow put this into a homogeneous representation. You cannot really use the classical named entity recognition tools for chemicals because they will take all chemicals and not only the drugs. So they also take metabolites, they take even water, they take CO2, and so on and so forth. Whatever they find that looks like a chemical in the text will be tagged. But here we are only interested into something that you can actually prescribe. And another problem is when you do this for database integration is that it must be extremely fast. So the typical chemical name integration works on an abstract, and you do this at the abstract level. So it can take a few seconds for an abstract. But here, we do database normalization. So we have a list of hundreds of thousands of drug names. And we have to all normalize them when we integrate this into our database. Yeah, and we did a normalization pipeline. So we implemented something based on different similarity measures. Then we wanted to evaluate it. Of course, again, we need a gold standard. So a list of different variations of naming a drug and the actual drug that we want to map it to. So we built them with a few hundred samples here. And then we did an evaluation and we see that our drug normalization pipeline works quite well. So F1 is here, the gray one, we are at 0.95. And we also tested the drug normalization pipeline of one of our competitors that's called MetaKB, and we see that at least our drug normalization currently works much better than theirs. Drug normalization, entity normalization is really something that people very often forget when they build integrated databases, and most of the papers that you find from a very applied perspective will have a normalization, but they all fail to do a systematic evaluation of the normalization. And I think this is really, really very important. You, you have to evaluate how well you do this integration. And here we contributed to this, to this problem. Yeah, we then, we built this database and then we started to look into this and in the light of a set of, of real patients. So there was a paper in 2019 where we had a set of roughly 120, the number comes later, of patients that were actually treated in the MTB of the Charité. And now we, for each of these patients, we had the cancer type, we had the list of variants, and we had a list of recommendations that were raised within the MTB. So it's, it's not really a gold standard, it's kind of a pre-gold standard, because these were the recommendations that were discussed within the MTB, but we did at this stage not record the actual decision. So it's more like a set of candidate drugs for this variant in this cancer. It's not the one that the MTB at the end really decided, okay, we go this way. Now we were asking ourselves, okay, how much could this process that was done all manually, how much could we have supported it with our iVis database? So we looked at all the different cases and then we had always the variants and we had the affected genes. Then we were starting to see how many precision oncology databases would you actually need to cover this set of variants in this set of patients. And we found that most of the genes and variants that were mentioned appear in only one precision oncology knowledge base. That means whatever you take, if you take only one, you lose a lot of information. Information. So here you have a lot in CGC and only in CGC, so 369 here, you have a lot in OncoKB and only in OncoKB, you have a lot in CIViC and only in CIViC. And then here you see the variants and the genes that are actually in more than— right here you see the genes that are actually mentioned in more than one of these knowledge bases. At the variant level, the overlap is even much smaller. Of course, there are many, many more variants than there are genes. And even at the drug level where you could think, okay, there are not, there are not thousands of different anti-cancer drugs, but even at the drug level, you find many of the drugs being mentioned in only one database. So here, for instance, CIViC has a very broad variety, mentions a very broad variety of drugs, while the number is much smaller in the database slide. OncoKp. Yeah, then we looked at the real cases. So that was what would we find in single knowledge bases. Is it really worth it to look into different? Then we looked at the individual patients and asked ourselves, okay, for these 113 real cases, if we use only one knowledge base, or if we use two knowledge bases, or three knowledge bases, for how many of them would we actually have any suggestion. So not the one that was targeted in the, in the MTB session, but for how many of them would we be able to come up with at least one or two entries saying, okay, for this variant, please look into this database, because here they found the following information. We also did a validation with a second dataset that is much smaller that was published from, from the Tim Byspaert lab in 2018, but there was only 12 patients, I think. But results are very similar in the paper. You find both of them. We find that for almost all of these patients, we found some suggestion in at least one of the precision oncology knowledge bases. So for, for almost all patients, it would have been worth to look into PoKBs. And in most of the cases, we would have to look into all PoKBs to really find all information. Now, if you would use something like a quality filter and say, okay, I think that information is more valuable if it's contained in more than one of the personal knowledge bases, then this would reduce the amount of recommendations already quite a lot. So here you see, these are the number of associations that we find when we require appearance in only one PoKB. So on average, something like 15. But if we require appearance in two, we are already down to 5. If we require appearance in all 4 of them, then we are down to 1 or something on average. And then there will be many, many cases where we don't produce a single association. Yeah, and then we really looked into the, into the, into the actual candidates for drugs that were discussed of these, for these patients. And we simply took the appearance of a gene-drug association in precision oncology knowledge base as an indicator for the relevance of this association. And then we built a classification model, actually a regression model. And we try to predict the drugs given a variant profile by their mentioning in the precision oncology knowledge base. So these, this is not at all a decision support system, because we only look at individual variants. Here. But it gives us an indication of which POKBs are important, and again, whether it's important and valuable to look into different POKBs. And this is what we find. So we get a good precision when we look only in one database. We can get the best recall when we look in all databases. And then there is a certain F1 measure where we look into a combination of 3 databases. I think, and then we get an F1 of roughly 43% at a certain precision and at a certain recall. And again, the main message here is it's absolutely necessary to look into different databases when you want to get the full picture. Yeah, this database, this iVIS database, we we don't implement it as a centralized database managed here where you would use a web interface for accessing this information, but we actually provide it as a framework that installs the database on your site. So we decided for going this way because we think that the information that we would that the queries that we would obtain are very often sensitive information because you give us information regarding the variants in certain patients. Of course, we don't know anything about the patient, but nevertheless, we know then about the variants. And we thought that it's much safer and much more secure if you, if people who want to use this actually can use it, can install it on their own system. And therefore we set up a webpage with, it's an installer When you install the system, it installs a MySQL database, it installs a web server, and then it starts downloading the databases and it integrates then the integration scripts for integrating all these databases, individual databases into this instance, and then it starts integrating. So we have a certain uptake. Currently there are 3 installations, one here, one in Göttingen, and there is one in a test I think in Heidelberg, actually, there are two more MTBs that are interested. We are going on with the evaluation at the Comprehensive Cancer Center of the Charité. So we are now working on a second evaluation where we actually know the final recommendation, not only the candidates. And there's also the International Variant Information Consortium. And we are in contact to them to maybe merge our efforts because they're building this VitaKB that I was just mentioning. And maybe we can throw the things together. At the moment, we ran out of funding. That's a problem. So we applied for new funding. We are looking for new projects. And then we want to— at the moment, we are only maintaining the system, but we, of course, have many ideas how we could further improve it.

Just a quick note, we are also approaching the end of—

Yes. So I— what is approaching?

So we have exceeded 45 minutes already by a couple of minutes.

Okay, then I simply, I would suggest I skip the third topic. The third topic would be on, so if you're interested, and I think I will provide the slides on text mining, especially text mining of German medical documents. That's a research area in its infancy because there are simply no German medical documents available. So nobody can really work on this subject that who is not him or herself a member of a clinic. And that is a real obstacle to research in this field. And we built the first openly available corpus for this. It's called Bronco. And I would give you many information about Bronco here, but I ran out of time. Sorry. My acknowledgements, so all this was funded by the German Ministry of Research and Education. There was quite a large group of people working on this. These were collaborators from the University in Tübingen, from the University Clinic in Tübingen, from the Charité, and from my own group. And there's a list of publications, and I'm now open for questions.

Thanks a lot. Fantastic talk. I also really enjoyed your focus on evaluation and validation in each of your topics. So great to see that. Yeah, please. We had a question about the licensing. So the question is, how does one avoid the licensing problems with COSMIC if they use the IVAS tool in diagnostics?

What is this specific licensing problem that you are talking about?

I guess you were mentioning that not all of those databases are open.

Yeah. And what we are only into. Okay. I mean, there are different aspects. So the, the The databases that we are integrated are all, for all of them it's allowed to download them, let's say. It's, we are typically not allowed to distribute them. So we could not build an IVYS instance here and then copy the MySQL files for, to other people, because then we would redistribute these databases and that's what is forbidden. So we can either only keep them locally, then we would have to give everybody a nice web interface. And let's face it, we and my group are very bad at building nice web interfaces. So we rather don't do this. If you want to see a nice web interface, go to MetaKP. And the other option is that we say no, no, you have to install it yourself. And they are also very other benefits from doing this. And then our scripts will download the database from your site and install it on your site. So this is the way how we deal with this license issue.

So we have a follow-up question from the same person. So the interest was on text mining and It's commented here that COSMIC is allowed to use for free only for research and not for diagnostics.

Yeah. So I have no answer to this question except from we are doing basic research, but this is not ein Medizinprodukt. Of course, we cannot, we do not claim, and I think we have disclaimers at various places, that this is a tool that you should today use for diagnostic and where you can write in your report, I took the following decision because IBIS recommended it. So this is, this is not the type of system that we are building. We are not a company. So I would If COSMIC would approach me and say, you're building tools for diagnostics, I would always say, no, no, this is basic research in tools for diagnostics. This is not a tool for diagnostics. So we are not MolecularMatch. We are, we are not selling reports about patients where we would run into these systems, or Molecular Health, which also offers this as a service. But I'm, I personally think there's a certain gray, gray area. I'm not a lawyer, but I hear terrible things about upcoming European regulations regarding research in medical, in the medical field, where some people say that in the future, whatever paper you write, you have to make sure that this is compliant to certain regulations regarding medizinprodukte. And that will be the end of all basic research in this field. I don't know how this will end.

So we have another question here by Jonas Born. How do you extract high-dimensional data? Is it possible to extract the data directly as a Multi-Assay Experiment object, or do you have just an API to do that?

High-dimensional data. So let's say the— or what I talked about here, nothing of this is classical high-dimensional omics data. So in the first part, when I talked about the search engine, of course, we have high-dimensional representation of text. I don't think that this is what you mean. In the second part, the iVIS integrated database is not the classical high-dimensional data. It's essentially a few thousand associations between genes or variants, cancer types, and drugs, with often with annotation and often with a link to the database where we have integrated it from and with a link to the literature. Where the database where it was originally created in, say, the actual evidence is contained. And this, so if you install IVIS, you have it all in a database. It's a relational database, and then you can do whatever you want. And there's also an API for searching it, but we have yet no particular export options.

Okay, thank you. I think that I should clarify the question. Tim Adler wrote, thank you for the very nice talk. You talked about high inter-expert variability with regard to clinical relevance.

Yeah.

What would you expect to see if you could have a look at the intra-expert variability? So would one person give you the same values if you asked the person twice?

I don't know, no idea.

Interesting.

Well, if you ask once and then 5 minutes later again, I think you will get the same answer. If you ask once and then a year later, you might not get the same answer because this is a field that is extremely fast, fastly, rapidly developing, and there might be a lot of new information. But let's say from our experience, you have the problem at the inter-level within an MTB, maybe not within one person, but within one MTB and at the intra-level between different MTBs. And it's really a big problem. So it's not small deviations, no, it's just completely different recommendations. Maybe it changed today. So the data that I gave you is now 2 years old. I don't know. That's speculation. This is— I mean, in Germany, there's more and more efforts to try to bring MTBs together and to work together and to have a more reliable and reproducible way of deriving recommendations. But as much as I know, it's not yet at the stage that you would expect similar recommendations when you go to, when you go to different clinics. I think if you have a rare cancer, it's really, it depends a lot on the place where you go for your treatment. And I would always say go to a university clinic, then you at least have an MTP, because otherwise you don't even have this option for looking into molecular data. At the same time, I mean, I must be cautious here. So we have very— so there have been a few studies trying to really look at the end-to-end performance gain that you have by personalized treatment in cancer. And these studies were not very— did not come to a very positive result. So it's really a field that is growing and growing and growing. And everybody— I think that everybody thinks this will be the future, but at the moment, If you look at something like median time of survival of patients who go through an MTB, through a molecular characterization, and you who then have a recommendation based on targeted therapies, then the differences are not significant. So it's really— I also am totally convinced that this is the future, but we are not yet there, let's say. And there's a lot of there are a lot of trials lacking where this kind of technology would be really evaluated in an end-to-end medical setting. And these trials are very, very difficult because they deal only with rare cases. Let's say when you look at the few trials that have been done here, their main problem is that most of the patients simply die before they even start the treatment. Because these rare cases that where there's no clinical guideline anymore, they are really, they have very, very severe cases. And so there's no time anymore to look at the result of a treatment. So that makes it very difficult. And you never have enough of one cancer type that you could really make a robust statement about a certain cancer type. It's always a very, very heterogeneous mixture of patients.

Yeah, thanks a lot for these insights. I also found It's striking to see the differences and the way you also evaluated it just by asking different teams. So pity that not more teams agreed to open their decisions. Would be really nice to see some follow-up research on that.

Yeah, but so when we talk to our MTB, I mean, one reason is because they all use PubMed, they simply spend 2 or 3 hours for every patient. So it's just an enormous amount of time that they have to invest. And that might be one of the reasons that many MTPs said, no, sorry, we don't have this for patients. We don't have this 12 hours to invest.

Maybe. Okay, so thanks a lot again for these insights into your research. Thanks to all of you for coming. Yes. And yeah, I think, Doreen, we are approaching the end, right?

Yes, yes, we are a bit over time, but we started a bit late, so sorry for this. Yeah, so thank you for this great talk, and thank you for the audience to be here. And we are at the end today. Stay healthy, everyone. Thank you for listening.

If there are any questions left, just send me an email. Thanks for the invitation again. Bye-bye.