Scientist Stories: Phillip Richter-Pechanski, Natural Language Processing in the Clinic

Welcome to the Data Science Seminar. We are very proud to announce the 25th edition of the Data Science Seminar today. It's my great pleasure to welcome Philipp Richter-Piszczanski. He has strong linguistic background and works as researcher and domain expert in natural language processing and clinical data science in the group of Christoph Diederichs at the Klaus Tschirra Institute for Integrative Computational Cardiology and the Cardiology Department at the University Hospital Heidelberg. Today, Philipp will talk about natural language processing in the clinical domain. The stage is yours.

Yeah, thank you very much, Doreen, for the kind introduction. Yeah, I'm very excited today to take part in this data science sessions. And I want to give you an introduction to the domain of natural language processing in clinical routine. Let me share my screen. Yeah. So the topic of today will be, as I already said, natural language processing. So we will look at the data format text. And in today's presentation, I want to give you an overview about our current projects. And at the end, I want to show you as well some concrete results we actually achieved using currently state-of-the-art NLP methods. Before I start, I want to give you a quick overview about today's presentation. So after a short introduction, and some important definitions. We will have a closer look at the data we've been using. Then I will outline our used methods, and we will define actually our final objectives where I will show you the results. And after that, we will sum up, and I will present you as well current challenges in NLP and our future work plans to solve these. So while structured reporting in the healthcare domain is an emerging field, Still, most of the data is stored in unstructured format. Natural language processing tries to extract relevant clinical information from these texts, and we then finally want to store it in a structured format so that clinical routine and research actually can process these data in their pipelines. In Heidelberg, we therefore focus on clinical entity recognition and text summarization. Basically for data preparation and clinical information tasks. So now I want to give you some short overview about our current projects we are doing at the DITILICH lab. So we are— yeah, we've seen a huge progress in NLP, especially with the rise of deep learning in the last decade. But if we compare it to the domain of clinical NLP, And especially if we have a look at the German clinical NLP, and we see that this is still in infancy, this domain. And the reason is because experts actually just have limited access to the clinical text data. And because it contains patient, sensitive patient information, and we have to always follow data protection regulations. This is the reason why in Heidelberg we intensified research on computer-aided de-identification, where our goal is to find patient-sensitive information in the letters and then pseudonymize or blind these information out, which are not then needed for any further tasks like clinical information extraction. We as well presented two of our approaches at recent conferences. And in addition, we are focusing on clinical information extractions, such as cardiovascular concept extraction. And here we focused on creating datasets and training machine learning models for automatic extraction, concept extraction. And here we focused on medication information cardiovascular risk factors, and anamnesis information. In parallel, we are developing a DFG-funded MeDEEP framework to rapidly create training data for these kind of methods. And here we are currently conducting projects for annotation of— with cardiovascular concepts, medication extraction, and document segmentation. Today, we want to focus on two methodologically very closely related tasks. So this is on the one hand de-identification and on the other hand cardiovascular concept extraction. So both tasks are actually part of so-called named entity recognition, and here our goal is to seek and classify phrases containing like patient names or patient addresses or clinical entities like drug names or any kind of cardiovascular concepts. In case of de-identification, we are focusing on the part of seeking and classifying patient data in the texts. And these are so-called patient health information tokens. And we are using or we are basing our work on the US Data Protection Regulation because in comparison to the European GDPR, this has a much better and more precise definition of when we can consider clinical text de-identified. And as well, as you've seen already, we are using the term de-identification. So this is like the most general term. And you will very often hear as well anonymization or pseudonymization. Very often they are used synonymously, but Actually, they are kind of different concepts. So our technical approach, what we will show here today, is generic, and thus it can be adapted to the European GDPR with legal advice. On the right-hand side, we are focusing on cardiovascular concepts, our second task. And here we are— we want to extract 12 cardiovascular concepts we selected with our clinical partners. So something like angina pectoris, dyspnoea, syncope, nicturia, or diabetes mellitus. So to get a bit more concrete, let us have a look at some snippets I just created, representative for our discharge letters. So in case of de-identification, you see here we want to, for example, seek the sequence Max Mustermann and classify it with a PHI tag so personal health class person. And our clinical use case is here, as I already said, that we want to make the data more accessible for NLP and deep learning experts. And thus, we want to first remove these personal health information. On the other hand, we have cardiovascular concept extraction. And here, you can see, for example, that we want to seek the sequence starke Druckschmerzen auf der Brust, and when it classifies, then with the cardiovascular concept angina pectoris. And our clinical use case would be here, for example, that if we would— if we are able to annotate discharge letters with these kinds of concepts, we could, for example, build cohorts. So we could ask the dataset, give me all patients which are suffering angina pectoris and are smokers and at the same time have the risk factor hypertonia. So before I will continue showing you the deep learning methods we've applied, let us first have a short look at the term supervised learning, which you will mostly already know, but in context of NLP, I want to show you just a quick example how this can work if you are working with text. So on the On the left side, you see a text snippet containing like a sequence of texts. So starke Druckschmerzen auf der Brust, 5 tokens. And here we are meaning implicitly angina pectoris. And we take this as our input data X, and then we input this into a machine learning model. This is our feedforward process. So this model is now at the beginning just randomly initialized with some kind of parameters. And this doesn't matter. It is not trained before. And we just input that now. And then finally, we will get for each word some kind of a confidence score and cardiovascular concept prediction. So as you can see here for the 3 first words, so starke Druckschmerzen auf, the model is already quite confident that these words are belonging to the concept of angina pectoris. Less confident is with the words der Bus. Here we see between 14% and 70%. And so what we need to do now is we need to optimize the model's parameters, and this we are doing by the backpropagation process. So we go back into the model and we are using optimization theory here, and practically this is very often gradient descent, for example. A very famous method here And we are now adapting the parameters in the model depending on the error rate of the predictions. So if the error rate is larger here, for example, like for the last two tokens, we have to adapt a bit more than for the other tokens, of course. So we can see here that we can now do this kind of process iteratively as long as we are actually achieving kind of a sufficient optimum for us. But it doesn't matter actually now what kind of model we are using here in the middle. The most important thing is here what I want to show you, that we need, if you have a look at the right-hand table, the second column which contains the labels. And these labels are needed for the model to calculate an error rate or a confidence score. And this means that These supervised machine learning methods can be just applied on clinical texts if we have annotated text available. So in this case, we need a discharge letter where a clinical domain expert annotated each word with these kinds of concepts. So this is called manual annotation, and this is a critical part in any kind of deep learning training in a supervised manner. So let us have a look at the data now. In total, we have at the cardiology department around 200,000 discharge letters, and they cover a time period between 2004 and 2020. These letters we find in a clinical information system, and they are typically stored in a Microsoft Word format. In case of de-identification, we chose stratified sampling to select 113 letters of these of this big corpus. And then we let clinical experts annotate these letters with personal health information like patient names and birth dates. And in total, they contain 110,000 tokens or words, and 5,200 of these tokens contained actual patient information. On the right-hand side, you can see our cardiovascular concept extraction dataset. Here we chose 204 discharge letters, and we selected just the anamnesis and the cardiovascular risk factor section in these letters because here physicians did the annotations and they, yeah, of course had a lot of time restrictions. So in total they contained around 380,000 tokens and just 4,000 of them are actually containing cardiovascular concepts. So let's have a closer look into the annotations here for our cardiovascular concept dataset. So you can see that in total, around 1,600 concepts had been annotated by the annotators. So here we are talking about concepts as such a concept can— such an annotation can contain several tokens. So it can be 1 token, 2 tokens, 3 tokens. So in total, we have 1,600 at all concepts.. And mostly they annotated dyspnoea and angina pectoris here typically in the anamnesis sections of the discharge letters. And if we have a look at the right-hand side on the histogram, you see that approximately 8 to 9 cardiovascular concepts have been annotated per letter. So now let's have a look at a very interesting attribute, especially for these kinds of datasets. So you can see here kind of a sparse heat map, let's say, and on the x-axis you see the amount of tokens. So from 1 token to 25 tokens, and on the y-axis you see the cardiovascular concepts, the 12 concepts we chose. And then you see in the color scale the distribution, the length distribution per concept. So let's take an example. If you have a look at the angina pectoris concept, you see that, for example, 52% of the annotated concepts contained 2 words. So most probably this will be explicit annotations. So there will be angina pectoris in the letter or pektangeöse Beschwerden or something like that. But as well, you can see that we have a high variance in the length of this concept especially, but as well, if you have a look at dyspnoe or edema or palpitation.. And this means that this angina pectoris can be a quite long implicit description. And we can have a look now here. I just chose some representative examples of these annotations. So on the one hand, you see shorter annotations like typische AP, so this is an abbreviation for angina pectoris, or 3 words like angina pectoris Beschwerden. And on the other hand, we have quite implicit and long contexts here, which the physicians annotated because they implicitly read, okay, this patient suffers angina pectoris, like for example, the sequence, Druck auf der Brust verspürt zu haben, welcher in den Rücken ausstrahlt und mehrere Stunden angedauert habe. So the challenge for a machine learning model is now to extract actually the semantical information behind these words and assign it correctly to the concept of angina pectoris. So let me give you now a short overview about our methods. So on the first line, we of course wanted to compare these deep learning methods to already well-established methods we are using in text processing, NLP, already. So I don't want to go into the details, but this is basically Our baseline is based on a conditional random field. So this is a statistical machine learning model. It considers the context of each given input token, which is very important in NLP. And it uses a so-called predefined set of feature functions. So we need feature engineering. We need to define these feature functions to run such a statistical machine learning model. And finally, it is trained using log likelihood. To train the weights for each feature function. So how important are these feature functions? A possible feature function would be, does this word contain a capital letter, for example, which would be important if we are searching person patient names, for example, because they are most probably written with a capital letter in the beginning. Now let's take a step back. In NLP, actually, the most important and very crucial part is representation learning. So what does that mean? First of all, if we are processing text, we need to convert this text in kind of a numerical representation. Otherwise, we cannot process it in our deep learning models because they are based on calculations and numerical inputs. And this is a reason why we just have to find a way to represent any word in a numerical format. So this is typically done in a kind of a vector format. And if we look at machine learning models, statistical machine learning models like the support vector machine, or what we've just seen, the conditional random field, we are using so-called feature vectors to represent a word in our case now. So we are inputting a word and then we are creating such a feature vector which has just the dimension of all the features we are defining. So it depends on us how many features we are defining. Let's say you define 10 features, so the vector will have the size dimension 10, and then you are just assigning to each of the dimensions in the vector the feature function output. So for example, if you input here a word which contains digits, so yeah, kind of a simple feature, we will now in the first position assign this feature, and then we will flip here if we take it in a binary format this to a 1. And we do that for all the rest of the features. And finally, we will have a feature vector in a numerical representation as a representation of the input word. So on the other hand, especially with the rise of deep learning in its early days, as well, they appeared another very important representation for a word, which is called a one-hot vector representation. So what does that mean? So basically, deep learning models, especially in the beginning, the idea was that a vanilla model, actually, you don't do any pre-assumptions. So the model itself should extract the features it is using or it considers important. So this is very different to the first representation where we've seen where a human actually defined features where the model should focus on, and the deep learning model should do it on its own. So we needed to find a way to represent the words just that as less as possible pre-assumptions are in it. And here we have the one-hot vector representation. And these are created like that. So we have a vector again with the dimension of the vocabulary size. And this you can get if you look at your corpus you want to apply your model on. And then you just count the unique instances of any kind of words which appear in this model. And the amount of words, of unique words, is then the dimension of this one-hot vector representation. So you can imagine this can be quite large. And so here, this is the dimensions. And just for example, let's say the word nicotine consume, you assign to the first dimension in the vector. So you flip this to a 1, and the rest of the entries of this whole vector will be 0. And so on for each word, you will have then such a vector with just one dimension with a 1, and the rest will be zeroed out.. And now you can think about that some disadvantages appear in here. So first of all, you have a really sparse representation. I mean, if you can think about vocabularies, they easily have something like 10,000 or even 100,000 or even millions of unique words. So you will have a huge vector representation, a huge input vector, which will make your model as well larger because already the input layer is so big. And on the other hand, You don't have any kind of semantical, syntactical, or contextual information in these kinds of representations. Because think about them in a vector space. So you will see that all the vectors are equally distant apart. So you cannot see any kind of similarities between words which could appear. For example, if you focus on fruits, orange and apple should have a vector representation which is maybe more similar than orange and a computer. The word, the vector for it. So this is all not possible with this one-hot vector representation. So let us have a look now at a very groundbreaking publication which has been done by Bengio in 2003. So we are talking about word embeddings now, which some of you maybe had heard of already. So here we are trying to represent words in a distributional representation. So we are going away from the sparse representation where we actually just have one dimension with a non-zero value, and we want to decrease the amount of the dimension of the vector. And here our word embedding, so let's say we have here a word dimension, a vector dimension of let's say 150 or 300. We want to now optimize and we want to use all the dimension entries in the vector for this representation. So let us have a short look at how these word embeddings have been actually created, because then you can maybe get an easier understanding of it. So I am just choosing now one of the possibilities how you can create word embeddings. There are several possibilities. Just to get an impression on it, let's focus on the Skipgram method. So, and here you can see we have a neural network with 3 layers. So we have an input layer, we have a projection layer and hidden layer, and we have the output layer. And our objective is now to train such a model in an unsupervised manner, thus we don't need annotations. So we just train it on a huge amount of text. So this could be, for example, the whole Wikipedia in English or in German. So we input every word into this model now And then we represent in the beginning, in the input layer, each word in a one-hot vector representation, as I already showed you. So this is a large vector with just one dimension representing the word. And now we are projecting, we're making a linear projection of this large one-hot vector representation into a lower dimensional space. So here we have our V. So this is like the vocabulary size. So it can be something like 10,000 or 20,000. And then we are just linearly projecting it to a lower dimensional space, the dimension of our hidden layer. So this projection layer here, so this can be, let's say, 150 or 300. In this layer, then we don't additionally do anything. So this is just this projection. And here, typically in neural networks, you would apply some nonlinear function like, I don't know, sigmoid, some activation functions. But this is not done here. In the next step, we take these projections and linearly project them again to the output layer. And here again, this dimension of the output layer is the size of such a one-hot vector. So the reason is here we are applying a so-called softmax function. So we have a vector, and in all the dimensions we have a distribution, a probability distribution. And we can see now where on the position where the probability is the highest, this is the predicted word of the network.. And this we are doing all the time now in an iterative process again. And finally, if this model is trained sufficiently, we will just use this middle projection because here in this 300-dimensional or 150-dimensional layer, our word vectors will be stored. So they are appearing in this projection. So we are cutting that out and we don't use anymore actually the rest of the network. So this was kind of a fascinating idea which bends your head actually to train such a model and then just to cut out a part of it and here your models are appearing, your word embeddings are appearing. So this kind of representation of course helped a lot. And there are very famous applications of it. So it was possible to now have these vectors which are semantically closely related and you will have X-valve vector representations, which you will see in a vector space closely related related to each other. And this, of course, made it easier for a deep learning model which uses these word vectors, these word embeddings as representation for any kind of training. So it already has some knowledge about the word. So it is implicitly in this embedding, some semantical knowledge is stored. So of course, we had some disadvantages. So think about some homonyms, so words which can have different meanings. So in English, this can be the word "can" or the word "rose," where you can think about you have just one word embedding representation for it. So there won't be two. So you cannot distinguish different meanings of these words. So you can just input them always in the same style. And in NLP, we have a very important saying. It says, "The word is characterized by the company it keeps." And this is a very important saying because it actually says everything about language and how language is working in a way. So you always have to think about the context. There are even languages existing where you, for example, don't use any vowels. So you just have consonants as words. So for example, in the Semitic languages like Arabic or Hebrew, and here you always have to think in the context and then you can fill in the vowels by yourself. And this makes very easily clear how important this context is actually for or a meaning of a word. And here now in 2017, we've seen another groundbreaking development, which really changed a lot in how NLP is working now. And the idea was we have seen the rise of so-called pre-trained language models. So what does it mean? So very important instances or famous instances are here, for example, BERT or ELMo, or the GPT model. And such a language model tries to involve context into account. So instead of representing a word now as a single word embedding, we use the whole deep learning model for representing an input sequence. So you always input a whole input sequence into this model. And then the model assigns to each of the input words a representation, a numerical representation.. But it does that by looking at the context always. So not just by the single word, but always look what are the rest of the input words. So, and this of course makes language models quite complex. So you can see here on the right-hand side a representation where some language models are listed here, like the DistilBERT or the BERT-Base, BERT-Large. Don't be confused with the words now. So these are just different architectures of pre-training such a language model. And you can see here some kind of an exponential development lately if you have a look at the amount of learnable parameters. So always you have such a deep learning architecture with randomly initialized parameters, and your goal is to learn these parameters that they are fitting to your dataset. And of course, if we have a look at the GPT-3 model, for example, which became quite famous because it, yeah, for example, is able to write on its own some journalistic articles about the quality you can discuss, of course, but it is kind of impressive what it is able to do. And here we are having already 170-something billion learnable parameters. So you can think about in our clinical NLP infrastructure, we most probably won't be able to train such models anymore. But that was not important because If you already have a look at these like hundreds of millions learnable parameters here, like the BERT model, these models already outperformed in other NLP tasks state-of-the-art results. So our goal was now to evaluate such a BERT model on our clinical data. So if we chose here, yeah, so easy to assess and a famous BERT model, this is developed by Google. In 2018, and this is based on the architecture, the deep learning architecture, it's called Transformers, which became very famous recently in NLP, but slowly as well in different domains you will find these kind of architectures. The most recently as well, the GANs, so this Generative Adversarial Networks, and they are as well using Transformers now. Very often. So previously in NLP, we used so-called recurrent neural networks. So here we have a sequential processing of the input. So this neural network took always one word after another as input. And the transformer is a bag-of-words approach. So here we are inputting the words in parallel and as well processing the words in parallel. And to do that, such a transformer has a method of so-called self-attention. So self-attention is to compute the similarity scores between the words. So what could be an application? Think about coreference resolution. So here our goal is to find out coreferences in a sentence and to which nouns they are actually looking at. So for example, you have the sentence, "Jim lives in Hamburg. He is 20 years old." Then our goal is to find out what this "he" is actually pointing to. And this is obviously pointing to the noun Jim.. And this self-attention helps you to understand these kinds of relations between the words. On the other hand, we want to train positional embeddings. So this is kind of a way how this transformer tries to again understand the word order. As I already said, this is not a sequential input, it's a parallel input. It just got all the words and the order is not important at the beginning. And now this transformer model uses the so-called positional embeddings to get this order into the input sequence again. So what is very important, so as I already said, the rise of these pre-trained language models, like for example BERT, really totally changed the way how NLP methods are applied, how they are working. Because yes, this pre-trained language models, this pre-trained is a very important term here. And we are having two distinct workflows now, the pre-training and the fine-tuning. Fine-tuning, I will explain you in a second, but the pre-training is working the following way. So you are having an unsupervised task. So you don't need annotated text. That's always nice. And because this is easier, text you have most probably available, but not manual annotations. So you take such an architecture now, randomly initialized again, like the BERT model, And as a dataset, you use, for example, the whole Wikipedia dump or something like that, whatever you have. And now you're training such a model with an objective of a language model. And very simply spoken, this is something like predict the next word given an input sequence. So if you have the input sequence, Joe Biden is, then the task of the model is to predict president, for example. So this is a language model objective. And this can be easily done. You don't need annotations here. You just have to input the whole text. And then the BERT model will always make another task, take some input sequence and then predict the next token. And we'll have a look if it is right or if it is not. And then the parameters will be adapted. So finally, after such a training process, you will have then a pre-trained language model trained with a BERT architecture which is basically understanding how Wikipedia language is working. This has to be said because, of course, the model learns the language of the input data. So if you give it, for example, in the next step, the Bible as an input, most probably it won't perform that well. And here now on the right-hand side, you have the next workflow, which is the fine-tuning part. So here we are having a supervised method So we again need annotated data. And this is a critical point here because this language model already has an idea of language, how language is working, what syntax is, what grammar is somehow working. And here we can use this knowledge. And because of that, we can have smaller training data because now on the training fine-tuning part, it can focus really on the task we are inputting it in. So for example, we are looking here at the text classification tasks. So our task is we want to input a sequence of words, aktiver Nikotinkonsum, aktuell 2 bis 3 Zigaretten pro Tag, and we want to classify such an input sequence into these 3 distinct classes here, non-smoker, ex-smoker, smoker. And so our goal is now we take the BERT model to represent all the words we are inputting now from our discharge letters. And then we stack on it a classifier, a simple classifier. It can have any kind of architecture, it doesn't matter so much. And here we are just training it now on our smaller training data where it has labels and annotations and it can learn from it now and to perform this kind of task. So important to remember here is always, or to keep in mind, is In NLP, with this language model training, we have a pre-training step and we have a supervised fine-tuning step. And the supervised fine-tuning step is typically the thing you will take most, you will spend most time with, because here you are applying your annotated texts and then train your classifier for your specific task, what you had in mind. So let us define now the objectives here, actually, to see some concrete results how this actually can work. These pre-trained language models for our clinical NLP texts. So the goal is to train and evaluate entity recognition models, so seeking and classifying phrases using German pre-trained language models. On the one hand, we have de-identification. So here we want to use a pre-trained German BERT model to fine-tune the model on the task to recognize PHI token in our German discharge letters. And on the other hand, we have cardiovascular concept extraction. So here we are using pre-trained BERT model to fine-tune the model on the task of cardiovascular concept extraction. So you see how handy it is actually. You can use such a BERT model now for your task, what you actually decide. So it can be the identification, but it can be as well cardiovascular concept extraction. That solely depends on the smaller annotated input data what we created. So let's have a look at results now. So as I already said, we are using a conditional random field, and here we are using that as a baseline to compare it to our pre-trained BERT model results. So in test sets, we are using a 4-fold cross-validation traditionally for de-identification and cardiovascular concept extraction. And as metrics, we are using the F-score, so the F1 score for cardiovascular concept extraction and F2 score for de-identification. F2 score or the F score in general is the harmonic mean between precision and recall. And the F2 score is giving more weight to the recall score. So the recall is here more important in de-identification. What does that mean? So we prefer to de-identify personal health information token— sorry, we prefer to de-identify a token which does not actually contain personal information instead of not de-identifying a token which contains personal health information. Yeah, so we want to have the recall as high as possible, and then it's okay for us to have some false positives where we actually have de-identified words which didn't contain patient information. So F-score in general is used for this entity recognition task very often because it handles this issue with unbalanced datasets.. If you remember in the introductory slides, you've seen that, for example, the dataset for cardiovascular concept extraction consisted of 380,000 words and just 4,000 are actually containing cardiovascular concepts. So we have to handle a lot of true negatives. And if we would, for example, apply the score of accuracy, you would easily see that, yeah, if you take a model which assigns to each word a negative label, this model will have most probably more than 99-point-something percent accuracy. But this is not sufficient for us, of course, because this is not the importance of seeing what you actually are looking for. So as well here, very important, always when you are defining your task, think about the scores you actually want to use for it because they have to be representative. Yeah, so what score you want to optimize? Yeah, so accuracy score will be most probably already perfect in the beginning without any training. So now let's have a look at the results. On the left-hand side, you see the results for the de-identification task. Here, I'm viewing the F2 score, and you can see that our BERT model over most classes outperforms our baseline already. If you have a look in more detail, especially for these classes here, person, location, or organization, you see that the BERT model performs quite well.. And you can see here that all these three classes are kind of variant, so they can contain anything. Yeah, so person names is like, oh, it's hard to create a list with all possible person names. And so it obviously looks like that our BERT model, which is so context sensitive, somehow performs here best. Yeah, so it's in comparison. So for example, for the salutation or for the zip code class, yes, the difference is not that big. So already our CRF quite well performs. Yeah, still BERT model is a bit better, but if you could, if you have a look at the significance here, then especially these classes are important for us now here. So, and if we have a look at the cardiovascular concept extraction, you see a similar behavior. So the conditional random field is outperformed by the BERT model over most classes, and especially in classes which are, yeah, what we've seen, angina pectoris, yeah, which is very long sequences sometimes, this implicit meaning. So somehow see how the BERT model here already uses its pre-knowledge about how language is working here to get such better results in comparison to the CRF. So how can we sum up these kinds of outcomes? So first of all, the less complex conditional random field, our baseline, achieves already promising results, though we use the small datasets. And yeah, you can— this conditional random field can be, because it's statistical machine learning, it can be easier to be interpreted. Yeah, and you can fastly train it. Such a model on our dataset, you can train on 20 seconds or something like that. So this is really fast. In comparison, such a BERT model is trained in like 1 or 1.5 hours with our IT infrastructure. Of course, this is depending on that. Still, our language model outperforms the baseline in de-identification and in cardiovascular constant extraction. Even on our small training datasets. And this is a very crucial point here because in earlier days, deep learning models always needed really, really huge annotated training data. Otherwise, you couldn't really apply them. And now with the use of this pre-training, you can now actually focus on your downstream task like the identification and create a specific dataset just for that. And it can be much smaller than it has to be, it had to be. So what you can see now, our hypothesis is as well, and that in this smaller training data as well as possible in the clinical domain now specifically, because our discharge letters are somehow semi-structured. So such a discharge letter always contains an introduction and very often contains a diagnosis section or an anamnesis section. And the cardiovascular risk factor section, a summary. They can be written a bit different and can be differently positioned, but somehow there is some semi-structure in it because the physicians are working with kind of a copy and paste workflow very often. Yeah. So, and of course, our models are using this kind of structure already and to remember basic contexts. So of course as well, we can say that these classifiers which are trained now on our local dataset most probably won't perform that well on other clinical datasets. If we have a look now at the clinics in, I don't know, in other cities or even in other departments. But here now, and it came, and you've heard already a very exciting talk about this federated learning. Yeah, so here we can get in touch with the possibility to train local models, yeah, and then if it is possible to distribute these models then via federated learning. So another point is that these local annotation projects are really more feasible. You can easier conduct such projects because you need less time, yeah, you have to Of course, in the beginning you have to activate or you have to find your annotators, you have to train them, but still it's possible to make annotations in, let's say, a few weeks and not a few months or even years if you are talking about 10,000 discharge letters or something. And this makes it more feasible just to conduct these projects. So what are the challenges? Of course, we have to say that still extending the dataset, for of course, would make the models more robust. So 113 letters is really not a lot, and most probably they will become as well old, you know, because the style how discharge letters are written changes over the years because the physicians are changing. And then it is helpful to add some letters to the dataset always, yeah, but you don't have to do that in such a huge amount that it is just not possible to fulfill. And another point is that our neural approaches are harder to interpret in comparison to conditional random fields or support vector machines. And this is like a black box, and there is vivid research here on how to interpret your machine learning models, because especially in the medical domain, in the clinical domain, this is a very important fact. Physicians want to know why this model made this and that. Prediction. And but here, yeah, we still have to do a lot of research because there is not an out-of-the-box solution existing so far. Yeah, just to come back to the data, yeah, because we've seen we have data-driven methods here. So the data, and a very famous deep learning expert, Andrew Ng— I am not always not sure how he is pronounced rightly— But he just said again, the community, the deep learning community, should focus more on the data in the next time. We created such great architectures, deep learning architectures, very complex. And we tuned the hyperparameters and everything on them. But actually, the most easiest part to improve the performance of your model is mostly by improving the input data.. So you have to focus on that. And this is the reason why we are conducting now in our Dieterich Lab here currently a framework. We are creating a framework, an interface. It's called MIDEAP, Medical Information Extraction Using Deep Learning. It's DFG funded. And our goal is to involve clinical experts more in this process because already in the data preparation part, And we need this clinical expert information because otherwise the model can just learn what it is inputted. And here we want to involve the clinicians into the preparation of datasets, into the annotation, and into the model optimization process. And to increase annotation speed, we want to improve, of course, the annotation, the way how the physicians or the clinical experts are annotating. And here we are using state-of-the-art technologies like data programming, weak supervision, or as well active learning. And another very important fact, and the reason why actually, yeah, clinical NLP, especially in the German area, is so still in its infancy, is not just the de-identification problem, but as well, I mean, of course, this is the reason of it, we don't have shared corpora. So the NLP community typically works like that. So somebody distributes a shared corpus, so it creates a corpus with some annotations and then just releases that to the community. And now different NLP teams can just build their models and then they can compare which model performs best. And we cannot do that until now in the clinical domain, but we want to change that. And our Goal is now to publish a dataset containing 500 German discharge letters from the cardiology domain to make it like of a performance corpus. So others can as well reproduce our results. So all this scientific best practices which are not possible till now can then be applied on these kinds of datasets. Yeah, so this is already the end of today's presentation. So I want to, of course, thank very much the valuable input I got from my colleagues in the Diedrich Lab. And on the other hand, what we've seen, we are dealing with data-driven methods. So our clinical partners are really very important here, and we have to involve them more. And here I namely want to thank Benjamin Meda, Nicolas Geiss, Ali Amr, Dominik Schwab, Christina Kriakau, and Sebastian Bender. Okay, that's the end of my talk.

Thank you very much. I'm looking forward to your questions. Thank you very much, Philipp. I hope you can hear me. Yes. Because I had some problems with the muting. Yeah, thank you very much again. It was a really impressive insight in your work, and it's, I think, really necessary work. So before we start with the question from the chat, I would also have a question.

So what changes could be implemented in the clinical documentation workflow to best facilitate NLP to medical data? Yeah, this is a very important question. And of course, yeah, here is a crucial part. So as I already said, currently we are working with this clinical information systems. They are very often kind of out of date. Yeah, so already since decades active. And then we have to deal with these Microsoft Word formats. And to improve this kind of API, so the possibility, the interface between our models and the data, we of course have— our goal would be to improve the representation of this data. So this can be a structured reporting. So we have here as well, especially in the HiMed Consortium, we are trying to implement such workflows, yeah, where already the physician is actually in a structured format somehow processing the information he's getting from the patient.

Okay, so we have a question from the chat. So it's, um, with pre-trained Word2Vec models. They are probably effective in text classification of mainstream language, but are they really effective for medical text classification? And the author writes, I tried Google's GenSim for classifying medical abstracts, but the similarity scores for medical vocabularies are not good enough. I found BioWordVec, which seems to be working well, but I think we need more pre-trained words to wake models for medical vocabularies.

Yeah, that's a very good question. Because, yeah, it involves this information what I just said when explaining the BERT model. And of course, these language models are learning on the text you input. So if you put Wikipedia input, then the model thinks that Wikipedia is German language, and this is representative. Of course, if you would input now Twitter tweets, you would most probably get some issues here. He's talking about now, or she is talking about a Word2Vec, so this is kind of a word embedding representation. So here we have to take a step back. These are not pre-trained language models, but pre-trained word embeddings. And here we have the same problem, of course, and I can fully confirm his results. Yeah, and if you are training a model on your word embeddings on a non-clinical domain, then most probably they won't perform that good. But here is an important— and so we did intensive experiments here. So for example, if we are looking— let's focus now on the language models. If we look at the pre-trained BERT model, this BERT model is trained on around 14 gigabytes of text, the German BERT model. Which you can download. Yeah, it's published from a startup here in Berlin, and you can just freely download it and use it for any kind of tasks. And this BERT model had seen really a lot of text, and our idea was now, okay, we have our 200,000 discharge letters, let's train the BERT model on this kind of text. So purely our discharge letters, and here we had around 2 gigabytes of free text. And we found out that the BERT model, which is publicly available, in both tests outperformed our locally trained discharge letter BERT model. And the reason is that it is still not enough language, not enough data. So 2 gigabytes is still not sufficient, obviously, for our BERT model to get really an understanding of the grammar. And here, really, a lot helps a lot. So if you have If you want to train such a BERT model, if you look, it contains like 150 million parameters to learn. So it really needs a lot of data to somehow adjust all these parameters properly. And we tried as well another approach. This is the so-called language model fine-tuning. So here we took the publicly available BERT model and gave it in addition to its knowledge it already had. And in addition, we inputted our discharge letters to it. And here, indeed, we've seen that at least this model somehow is comparable to the publicly available BERT model, but improvements had been slightly and just for some classes. And so not in general, we couldn't as well say here that this model outperforms the publicly available BERT model. Which is trained on any kind of data. This is basically trained on Wikipedia. I know it because they publish their work. And here, I mean, Wikipedia contains a lot of clinical texts as well. And there's a lot of cardiology texts. You see that when you have a look at the vocabulary, as I said, every deep learning model uses a vocabulary. So it knows a set of words. And if you input then words which it had never seen before,, it will not be able to represent it properly. So it has to impute something. But obviously in the cardiology domain, we didn't have such a huge problem with this unknown word things, because it has seen quite most of them we have been using in our cardiovascular documentations. But it's a very important step. And of course, the larger the corpus would be, so this 200,000 letters is not sufficient, but it doesn't mean that maybe if we would be able to get much more data, then most probably it would help. In the English NLP, clinical NLP section, we have such models already. So this is called BioBERT or ClinicalBERT, and there is a DischargeBERT even published. Yeah, so they've been— in the English clinical domain, you have these shared corpora.

Yeah, and so they've been able to train these models on publicly available data. Okay, thanks for the answer. So I would go on with the next question: What is the benefit of using deep learning for CCE compared to simply having a manually generated set of identification phrases that map to disease? And in other terms, how consistent is the description of diseases in doctoral letters if the variability of the description is low and the number of signal words is limited, it might be easier to create a translation map.

Could you please comment on this? Yes, yes, as well a very good question. So here we are coming to the point of as well the, yeah, I mean a doctoral letter is the way how you can, yeah, there are ways to interpret a doctoral letter because a human wrote these letters. And some are reading it a bit different. I had a very important, I had very interesting review rounds. So we had these physicians who annotated these kinds of concepts and we've got very interesting discussions between the physicians. And because the one said, no, I understand it a bit more like that. And the other understood it a bit more another way. So it was very interesting to, you know, we had to write guidelines for this annotations. So that both annotators are actually annotating the same style. And so here, of course, yeah, which this is always the thing if you are documenting in a free language manner, that there's always a way of interpretability. Yeah, so this is never a distinct information, and this is the reason why it is so hard to capture that by any kind of mappings or rules. Yeah, because Yeah, it is really high variance in it. And we try to solve this now with this deep learning models. I mean, this deep learning model does nothing else than trying to understand these two physicians, how they interpret cardiovascular concepts. And it does not do more, but it doesn't do as well less. Yeah, this is very interesting. It is just to, we want to find a way to get more robust models and the rule-based approaches or mapping approaches, til now just did not give us a sufficient result. Still, they are very important and I don't want to remove them from our pipeline. Yeah, still they are— this is like an addition just. Yeah, it is—

we want to combine these methods to make a strong prediction. Thank you. There is another question, so I would say if we could hurry up a bit, then we, we have 4 questions, um, in sum. So I would start and maybe we can go through this 4, then we can finish in time. So the question, it's from Miretu Kibede. I hope to pronounce it well. So how about In a heavily imbalanced dataset, my data is hugely imbalanced, 5% versus 95%. I used upsampling and downsampling strategies. The upsampling method results in overfitting. What is your opinion on this? How large could the data be to apply deep learning? Is 400 observations enough, 200 by 200, the loss by class labels? I mean for the training data. Maybe if the answer would take too long, we can also— yeah, I could connect you and you could share this answer by email or maybe by a direct contact if you want.

Yeah, indeed, it sounds like a very exciting experiment, and it is quite a complex answer I would like to give here, yeah, because it involves a lot of different information, yeah. So maybe that would be a way, but just shortly speaking, I mean, this imbalanced datasets. Yeah. So as I just presented with our named entity recognition, we have that very often. Yeah. This is not a specific problem actually. We have our technologies to cope with that. Yeah. So because in a text you want to identify patient names, so most probably most of the words won't be patient names. So we have really imbalance in it. And this is not from the beginning a problem for a deep learning model. Or even not a statistical machine learning model. It really heavily depends what you want to do. I've heard now numbers like 400 samples, it sounds more like text classification or something. I'm not sure, yes, this is why I would need to know that. And here, of course, you have to, yeah, that's getting a bit more problematic then because with such a low amount of training or test data, your results might be quite biased then. Yeah, and that's good possible. But yeah, indeed, it's hard to go into the details, you know.

But nice to hear that somebody is involved in that. Thank you also on this. So Tim Adler asked, do you have an explanation or hypothesis why the BERT model performs worse than CRF on the task of Familienanamnese in slide 20?

Yeah, that's a perfectly fitting question here. And indeed, the interpretation of why different concepts, actually the BERT model didn't give us any additional, better results. So in the case of Familienanamnese, I did look into the data and into the annotations, and this Familienanamnese is indeed kind of typically just a one-worder, Familienanamnese. So it's a very explicitly written, most most often in the discharge letters. And so our conditional random field actually is quite well finding that. Yeah, so we do in this specific concept, our huge deep learning approach was not like needed.

Yeah, in this case. Thank you. Then we have another question.

How much time is needed on average for an annotator to annotate a single discharge letter? Are the annotators doctors? Yeah, very good question. And we are trying to measure that always. Yeah, it's not that easy. But of course, it depends on the task of annotation. So what do they annotate? But in context of cardiovascular concept extraction, for example, indeed, two physicians annotated here. Because, yeah, you've seen this implicit information. So this is very helpful for us that physicians who actually write these letters are trying to annotate them. And here they took around 3 to 8 minutes per letter. That was very varying, yeah, because in some letters it's just less information than in other letters. But around this we ended up. But it depends heavily on the complexity of the annotation task. So this is as well important. Don't make it too complex, because if it's too complex, you most probably won't get high-quality annotations because annotators already are. Yeah, I mean, they are as well humans. Yeah, you always have to watch out.

Yeah, that it's clear and that they can concentrate on things. Thank you very much.

And the last question for today: Is it possible to use the methods described today to develop a clinical ontology? A clinical ontology? I mean, I'm not sure about the idea behind it. I mean, it's like a clinical ontology is a way of structured representation of clinical terms. Yeah. And you try and sometimes they are used to do some kind of mappings and this stuff. I mean, yeah, straightforward, that wouldn't be that easy. It's not like that's done for that. But of course, in like a workflow or a process, it can be involved, but that's going too far. So it would be a deeper discussion.

I hope somehow I got this question right, otherwise I'm sorry for that. Yes, otherwise I could send you the contact or the person could contact you directly. That would be the best way to do. All right. Yeah, we are at the end of our seminar today. Thank you very much for this impressive talk. And for the audience, Thank you also very much for listening, and you can later on watch our video on our YouTube channel. And yeah, thanks again, have a good time, and stay healthy, everyone. Goodbye. Goodbye.