Keynote Roman Klinger @CLIN31

All right. Welcome everybody to this presentation. Thank you very much for inviting me. It's a pleasure to talk about this topic to you, and I hope, given the different— the various topics that I've seen in the first session, my impression is also that there would be some— that there's some shared interest on the topic. So I would like to talk about how emotions are communicated in text and how psychological theories can help in Computational emotion analysis, and I also called this talk Showdown Talent. I'll tell you a little bit later why I used this phrase. I could also just have called this talk Appraisal theories from psychology for emotion analysis in text, which is shorter but less catchy. So I decided to just use this as a secondary title. These slides are available online. You can scan this QR code or just type the URL into your browser. And this is joint work with my PhD students Yevgeny, Laura, and Enrica, a couple of undergraduate students Felix, Jan, and Amelie, and my collaborators Sebastian and Kai. So emotion analysis— what is this for the case that you have not seen this before? The idea is that we would like to associate emotions to textual instances, to textual realizations. And it helps to get an understanding what this is about when we just look at some examples. So let's just do this. Let's look at a couple of sentences, and I would like to ask you to decide which emotion is associated to these instances. And at the same time, ask yourself how you recognize this. So what about this sentence here? I can see the chat, you can, you can answer in the chat. So which emotion is associated to this instance here? She became angry. Yes, so many of you answer anger. That wasn't too difficult. Some of you already mentioned— also mentioned that it's explicit in the text, and I agree. There's just the word angry in there, and it's pretty simple to recognize that this is about anger. What about this one? A tear is running down his face. Which emotion might be associated to this text? Sadness. Most of you say sadness. I also see somebody saying happiness. Or perhaps joy. Yeah, so it seems that you agree that it's either sadness or joy. And how did you recognize this? Probably by interpreting what you— what this event, right? So you interpret, you know that Crying is a reaction to an emotion, and you know that it's probably mostly about sadness, but luckily often also about joy. And yeah, so this is, this is how you interpreted this. So it's about a bodily reaction, which is unconsciousness, unconscious topic typically. What about this one? We are going for a walk at the beach. So which emotion is this, is felt by these people who are At the beach. Joy, happiness, perhaps neutral, relaxedness. Yeah, neutral. Many of you say neutral. Some of you say question mark. Some of you say happiness. So we see that it becomes a little bit less clear. It's not so, so easy anymore to assign an emotion. But some of you came up probably with some background story and made some interpretation about this event and then associated this perhaps with your, with your own experiences. Last example: I want to run away. Fear. Upset. Fear. Anger. Fear. Disgust. Fear. Sadness. Disgust. All right, so nobody says something like a positive emotion here, right? So desperation. Yeah, so it's It's definitely about a negative emotion which people try to avoid, and this desire to avoid a particular situation leads to a particular emotion and also leads to an action tendency. So, and this action tendency is now described here. So what we've seen now in these examples is that there are different ways to communicate an emotion, and I already mentioned some of the devices, and we will look into this a little bit closer. So what I did now is What I did now is I just— we discussed a couple of emotion classes, emotion categories, just because you came up with those. And we will look into this a little bit closer now. And after that, we will also discuss how emotions are expressed or can be expressed and how we can interpret this computationally. So if we want to come up with a set of emotions for a categorical system, let's say for a text classification system, We could just try to come up with one on our own, but of course we could also rely on psychological theories because psychologists came up with such categories. So now the question is, of course, what is an emotion? What is not an emotion? And Ekman pretty early, I think already in the '70s, but he talked about this also in papers in the '90s, he came up with a categorical system which is called basic emotions, and he decided— he defined a set of different properties of an emotion which needs to hold such that you can call an emotion a basic emotion. And there are some things in there— I'm not going through all of them— there's something there, some things in there which you might have seen. For instance, the distinctive physiology. So this is something that leads and led to quite a lot of depictions like this one, where people show their emotional expressions and you can pretty easily recognize those. If you Google for Ekman and fundamental or basic emotions, you find many of those. You might also find this one. which I made some time ago. But there are some other things in these definitions which I would like to highlight, and this is about events. Appraisal— emotions are caused by or have antecedent events, and there is also an automatic appraisal. So you need to evaluate the event, not necessarily— you don't perhaps know about this evaluation, it might be unconscious, But there's a cognitive evaluation of an event and you make decisions about this. And this is what we would like to model. I'll come back to this later. There are also other models in natural language processing which are used for emotion classification. And one which is also pretty popular in natural language processing is Plutchik's wheel. So this is a wheel which also puts these emotions into some structure. There is an intensity. So things that are more inside of the wheel are more intense, outside are less intense. There are mixtures and there's this opposing property. In psychology, this wheel is debated because you might say that there's not evidence for this structure in psychology, but in natural language processing, this is pretty— a pretty popular source for categories of emotions. So if you want to compare different emotions, there's also another model which is sometimes called an emotion model. And this is the model of affect. So affect is actually not an emotion, it's more fundamental experience, affective experience, and affect is typically modeled by some dimensions which include arousal and valence. Arousal means how activated am I, valence is how positive or negative do I feel something. And when you have this 2-dimensional system, you can— or coordinate system— you can put emotions into, you can place them into the space. So for instance, when you are very activated, arousal is high and valency is low, you feel angry. If the valency is positive, but perhaps you're not very activated, then you're satisfied, pleased, perhaps calm, perhaps relaxed. So this is not an emotion model, it's a model of affect, but this is also used in natural language processing. So what we want to do in emotion analysis is that we take some text— here I exemplify this with a tweet— We want to put this into an emotion analysis system, perhaps a classification system, and we would like to output some categorical emotion. This is interesting from a set of different perspectives. In computational linguistics and modeling, this is interesting because we can learn how emotions are expressed in language, how they occur in language. But it's also relevant for psychology or social sciences, and also, of course, computational social sciences, because we can better understand emotions and their effects. In society, but also for individuals. And there are many applications in humanities, social sciences, but also in pharmacovigilance or robotics and intelligent agents. So such examples for applications are, for instance, computational literary studies, a study which we did a couple of years ago where we took Project Gutenberg, the assigned genre which we see here on this for each row, and then we can plot the average curve of the development of emotions And we see that, for instance, just as an example, adventure stories typically end with a lot of trust. And we can then use this also for classification, automatic classification of genres. We were not the only people who did something like this. Dotz, for instance, looked into valence of song texts and he found— they found that gospel and soul is very positive, while metal and industrial music is not so much. And actually over time, the change is not too significant. So now I talked about what we want to do in emotion analysis, and I talked about emotion theories in psychology. So how can we bring these 2 things together? So this Ekman model clearly serves as a categorical system for text classification. The same does this Plutchik model. In addition, Plutchik's model of different relations between emotions has also been used to inform classification systems such that they can actually make use of these relations between emotions, and this model of affect can be used as a regression task. But then there's this other thing in there, which is the role of events and observable characteristics of events, which has been introduced by Ekman. So Emotions and events seem to be related and seem to be affecting each other. And I think they do in at least 2 ways. On the one side, emotions are events. So when you feel an emotion, this is an event. And on the other side, events cause emotions. So this is perhaps a circular relation between these 2 things. Emotions are events. When you think about events in natural language processing, you could come to frame semantics and then A sentence like Donald is happy about his birthday present could be analyzed with a semantic role labeling tool or with frame— with a FrameNet emotion-directed frame, and you recognize that there's an event triggered by the word happy here, that then there's an argument to this event which is the experience, and there's a stimulus which is his birthday present here. And this motivated the task of emotion semantic role labeling, which receives quite some attention, particularly in In Mandarin, actually, but also in English. But this is not what we want to talk about today. What I would like to talk about is about events that cause emotions. So let's say we have a textual description of an event, and there is a car on fire. Doesn't express an emotion, but it is relevant. It might be a relevant event for the speaker, and it might cause fear in the speaker or other emotions, but let's say it's fear. And But to infer this, this requires the interpretation of events to infer possible emotions. And this is what I would like to talk about today. This also brings me to this first phrase of the title of my talk, show, don't tell. This is a paradigm when you— for authors of texts like novels or poems or pieces of art. And this tells authors that they should let the reader experience this story through actions, thoughts, or senses and feelings. So a text of a higher quality does not directly mention emotion, but actually does describe bodily reactions, actions, events, and such things. And this also motivates us that we want to bring together event descriptions and emotions. So what we have is we have textual event descriptions, we have emotion categorizations, and we would like to link these 2 things with an NLP system, which is better understanding what the system does also linked to some psychological theory. So we need a psychological theory and we need a natural language processing system, and I will talk about both now. So luckily there are psychological theories which explain how emotions are linked with events, and these are appraisal theories. That's what I would like to briefly introduce now. So let's get started with a definition, with a clear definition of what an emotion is. So So Scherer defines an emotion as an episode— it's comparably short, it's an episode, it's not long— an episode of interrelated synchronized changes in the states of 5 organismic subsystems. So something happens in your organism, yeah, and you can separate this into 5 different subsystems. This happens in response to an evaluation of a stimulus event. So we have an event, this event is evaluated, And then in parallel, different things happen in your body. So what are these different things that happen? Let's say we have this event: a person sees a car burning. This person doesn't look very excited about this, but this is just because I didn't know how to draw an excited person. And now this person can develop various changes in organismic subsystems, and these are a subjective feeling, which is a subjective experience, an expression of an emotion— shouting, for instance. It could be a bodily symptom like starting to sweat when you are afraid, for instance. Could be an action tendency like running away. And it can also be a cognitive appraisal, an evaluation of this event, which then also influences the emotion which is developed. And we hypothesize that each of these different components, emotion components, can lead to linguistic realizations which are used to communicate an emotion. So I can tell you about my emotion by reporting my subjective feeling, I can express my emotion, for instance, by swearing, I can describe my bodily symptoms to you, I can tell you about my action tendencies, and I can also tell you about my cognitive appraisal. For instance, this situation is not in line with my goals, and then this should help me Or help you perhaps to understand my emotion. So now let's look into these cognitive appraisals a little bit more, because this is what cognitively links events to emotions. And it turns out that there is quite a set of various appraisal theories, one which is very specific for stress, and because it's so specific, it's a nice first look into appraisal theories. And it's also pretty old, actually. So what we have in this Lazarus model of stress, which I would consider an appraisal theory, we have a stimulus. This is the environment. He calls this environment, but it could also be seen as an event. And then there's a— then this is perceived by a person. It could also be filtered away, but let's say the person decides this is relevant. I don't fit— the perception doesn't filter this away. then the person comes up with a primary appraisal. So the person decides, is it positive or irrelevant? In this case, stress is not developed. But if it's somehow dangerous, the person needs to evaluate cognitively if they can deal with this, if there are sufficient resources to deal with this, or if not. And if there are not enough resources, then stress is developed. So you see that this is a very structured approach to emotions. It has not so much a subjective component, actually. So now this is just for stress. There are other appraisal theories, for instance, the one by Scherer, which is part of the Component Process Model, and he has, I would say, 4 main components. One is, is the event relevant? So does this event actually affect me or my social group? And sub-dimensions are, for instance, if it's novel, if it's pleasant, and also if it's relevant for my own goals. If it's not relevant for the goals, I probably don't care about it. If it's not relevant, perhaps novelty might still make me curious about it. It's also about implications. So are there consequences for my goals and well-being? And this also has something to— yeah, so this is mostly about the goal. It's also about the the causal attribution and estimating possible outcomes. There's also coping potential. Can I actually deal with the outcome? Can I adjust them? Can I just adjust the outcome? This is about being able to control the situation. And finally, there is a normative component. Is this in line with my standards? So this can be internal or external standards. So if the event is about something that I find socially not acceptable, morally not acceptable, then it doesn't matter so much if it actually affects me or my social group. I might just develop an emotion because of that. So another model— so this is the model where we will also look back to in a second, but there are other models which already have been used in natural language processing. And one which I would like to highlight is the OCC model of emotions, named by the developers Ortony, Klawe, and Collins. And they also look into these different subcomponents, like who's the feeler, and the feeler, the person who experiences the emotion, is linked to the world through emotions, and the world is represented by variables which then represent events, people, and objects. And other variables are, is it focused on the self or on others, and a component of time— is it in the past, the future, in the present. And the model that they developed actually suggests quite a set of rules how to put these different components together to infer an emotion. So just as one example, if the consequences of an event are negative, if I evaluate it like that, if I evaluate an event like this, if these are consequences for the self, and if it's a relevant event, and if it has not been confirmed yet, for instance, because it's in the future, it's expected to be in the future, then it's likely to develop fear because it's about something negative in the future that's relevant for me. Now, if it turns out that it's disconfirmed, doesn't happen, then I would develop relief because I know now that this bad event doesn't happen. So this rule I actually took from, not directly from the OCC model, I took this from papers which actually linked these variables in the OCC model to text to make event descriptions in text interpretable. There are 2 papers which are quite similar. This paper here, which is a little bit younger, has a more extensive evaluation of the model, while this paper here has a slightly more sophisticated methodology, but the idea is in principle the same. They develop quite they make use of quite standard natural language processing technology to estimate these variables in this appraisal model. So what they do is they essentially check for the time of an event description. So given a sentence, they run a syntax parser, part of speech tagger, and decide is it about the present, past, or future. To decide if it's directed on the self or on others, they use the dependency parser in one of the papers and the semantic role labeler in the other paper, and both seems to work well. And interestingly, this estimate if an event is— how an event is evaluated, this is quite straightforwardly estimated with dictionaries. So they use lexicons for event polarities. They have event dictionary in there, an action polarity dictionary, and an effect valence dictionary for objects. And I think it's noteworthy that these dictionaries, these lexicons, are just positive or negative associations. Into which emotion an event is categorized then follows from these rules which I just exemplified. So this is, I think, a pretty nice approach to link event descriptions to text because it's very— to emotions because it's quite understandable. But of course, one question you could come up with now is if there are perhaps more modern approaches of using machine learning to link emotions to text with the help of appraisal theories. And this is essentially what we tried in a couple of papers from last year and this year, and this is what I would like to talk about now. So what we have is we have a text, for instance, a tweet like this one, and we have an emotion analysis system. And normally what we would do is we would train a— some deep learning BERT-like transformer model or something like this. We would fine-tune this to do this categorization of an emotion. Now the question is if we could somehow inform this classifier about the appraisal theories and if this might help definitely the understanding of how emotions are expressed, but perhaps also increase the performance of these classifiers. So what we would like to do is we would like to develop a pipeline or a model which makes use of this component process model of emotions and particularly of appraisals. And then we would like to know if perhaps this route here, where we go from through the, through the system, through these theories, and to the categorization, if this perhaps works better than just end-to-end learning. And this is what we will look at now. So, and this is what we did in this paper from Kooling last year. So if we want to build a classifier, a machine learning based system, we need a corpus. So we need to annotate a corpus. So to annotate this corpus, we asked annotators to decide if an event description— in an event description, the writer was likely— did likely want to devote further attention to the event. So we call this attention. Was certain about what was happening. Had to expend mental or physical effort to deal with the situation. So it's somewhat exhausting, perhaps. Found that the event was pleasant. Felt responsible for the situation. Found that he or she was able to control the situation. And finally found that the event could not be influenced by anybody. So this is somehow an opposite of the control dimension, but it turned out that including this explicitly increased the internotator agreement. So now these dimensions are slightly different from the dimensions which I mentioned in the other models before, and these concepts follow a paper from 1985 by Smith and Ellsworth, who who looked into quite many emotions and quite many ways to measure appraisal, and then they did a principal component analysis and found that these are somehow the main components for appraisal which help to distinguish quite a large set of emotions. So if we want to build a corpus of event descriptions with these appraisal dimensions, we need a corpus of event descriptions. Such corpora exist. And one corpus is also one which we built in 2019, which has been created following typical strategies in psychology. So it's not a corpus of texts which is gathered somewhere, but it's a corpus of texts where people were asked to describe events. So we asked people in the crowdsourcing platform, please remember an event which triggered a particular emotion which we gave to them and describe it. So they needed to continue a sentence like, I felt anger when, if that happens or happened. Yeah, so we got roughly 1,000 event descriptions stratified by emotions here, and the examples look like this: I felt sadness when I saw a homeless cat on the street. I felt shame when someone commented that I was looking very untidy. I felt anger when the police did not update me on a crime. And of course, in a classification system, we won't show these words here to the classifier to make the decision. This is also a setup which we implemented in the shared task for WASA in 2018, but there we collected the data with some weak supervision in social media. This year, it's really quite high-quality data because people described the event in which they felt the emotion. So when you do this, you can analyze the corpus, of course. The corpus had already these emotion annotations. Now we can look into the link of these appraisal annotations to these emotions, and some things are very nice and some perhaps show some noise. Perhaps, but let's first point out the nice things. So for instance, what we see is that the difference between anger, which we have here, and the emotion of fear— so I should first say what we see here. So we see one radar plot for each emotion, and the dimensions in each radar plot are the appraisal dimensions, and they are actually counts of instances which we have in there. The corpus has been stratified by emotion. But not by appraisal. Okay. So, and when we look at, for instance, anger versus fear, we see that this dimension of certainty clearly makes a difference. And this totally makes sense psychologically. When I'm certain about what happened, I'm not afraid of it typically, I'm angry about it. So, for instance, if something is in the future, I'm typically not certain about it, and this allows me to develop fear. Another interesting dimension is pleasantness and joy. Obviously, joy is the only positive emotion in this set, and it's also the one which has highest amount of instances annotated with pleasantness. Sadness, for instance, is interesting because the circumstance is quite high in contrast to other dimensions. The person has not been able to change the situation, it just happened to them. These psychological make sense. I would also like to point out one which is perhaps a little bit less positive or perhaps less intuitive, which is the difference between shame and guilt. So often people don't really intuitively understand the difference between shame and guilt, and it seems that our annotators also mix them up every now and then. Typically, you can only develop guilt when you are responsible, when you feel responsible. Otherwise, you can feel ashamed. And this is something that we can see in the data. So this responsible dimension here, this is slightly higher than in shame. But typically, you, you would expect this to be a little bit higher. So it seems that there's some noise in there because of the annotation process. So now let's see if we can make use of this in a classifier. So we build a set of different classifiers that abstract away details. So we use a neural network which just takes the text and predicts the emotion. We also build a neural network which takes the text and predicts the appraisal dimensions to see how well this works. And then we put these 2 things together and we try to make use of the appraisal to first predict the appraisal and then based on the appraisal predict the emotion. Note that we don't give the model the possibility here— this is, this is an edge which doesn't exist. Yeah, so there's no way that the emotion model can access the text. It would probably make sense from a natural language processing perspective, but psychologically, it's more interesting to evaluate if we can only estimate the emotion from the appraisal. But we also wanted to, of course, look into both things together, and we built a multitask learning network here in this setup. This is a simple multi-head classifier. And the architecture of the neural networks is here a simple— I mean, it's not too simple, but it's an LSTM. It's a BiLSTM, I think, with GloVe embeddings. And this one here, this model up here, is a dense neural network which just takes the appraisal dimensions as features. So let's look what's happening. Can we predict appraisals from text? Well, yes, it seems we can. We don't really know what these numbers are worth. Because there are no other corpora for this, and there are also no other developments yet to develop such corpora. We did this here now with appraisal— I'm sorry, with an LSTM. If you use a BERT classifier, which we also did— this here is the LSTM— if you use BERT, the results are roughly 10% higher, but the results between the different dimensions don't change. So now the main question is, of course, if we can make use of this for emotion classification. So So let's compare this to a plain simple text classifier. So this is an LSTM. Now we only look at this black bar, which is an average of the different emotions, but I also show the individual results for the individual emotions. And now we would like to include the appraisal classifier into this model. So we first predict the appraisal and then we predict the emotion. So I sketch this here as text to appraisal to emotion. And unfortunately, we see that the performance goes down. So this could be because of noise in the annotation. It could be because the appraisal time, the classification of the appraisal is not good enough. Of course, this motivates the question if it's in principle, if it in principle makes sense to go this way and you can replace the prediction by the appraisal, of the appraisal by the annotation and the data. And if you do this, you can actually see that it's worth looking into appraisal. Just our appraisal dimension classifier is not good enough yet. You could also ask if— so this is of course not a real-world model, right, because we make use of the annotations, but it's nice to evaluate if the appraisal has information that's helpful to predict emotions. So now you could ask, okay, perhaps does this pipeline here, which first predicts the appraisal, does something else than to just just the emotion classifier. So we would like to know if these 2 models do something which is complementary. So what one can do is build an ensemble which is just informed by an oracle which of the 2 models will work better. And if we do this— so this is actually the model— when we do this with this oracle ensemble, we can see that the performance is actually a bit higher. So this is also not a real-world system because it knows which of these 2 systems here is better, but it shows that the information of the appraisal-based model and the non-appraisal-based model are complementary. I skipped the multitask model because it just mostly relies on the text information, yeah, so this doesn't really help. Okay, so that's nice. We know now that appraisals make sense, though we are not able yet to beat an end-to-end classifier, but you could also ask if Perhaps the emotion components are somehow helpful for further use. So we wanted to know if making use of emotion components, how the emotions are expressed in text for emotion classification, might help. So what we did is— so I'll answer your question later, Francois, okay? So what we did now is we annotated 2 corpora, a Twitter corpus and a literature corpus. For the way how the emotions are expressed in there. So these 2 corpora have slightly different sets of emotions, and one is a single-label task and another is a multi-label task. And we have— I mean, the corpus is not too big, but it's sufficient to do this analysis. So now let's look at a couple of examples such that you get an intuition what this corpus is about. So now I show you 2 examples from each dimension. One One is from Twitter and one is from literature. And you can guess which one is from which domain. So instances which are annotated as making use of a cognitive evaluation of an event to communicate an emotion are, I can't stop, or found my old lava lamp. So if you grew up in the '90s like I did, you know that finding a lava lamp is actually associated to a lot of joy. Events which, or emotions which are communicated by descriptions of the bodily reactions are, for instance, she did not know she trembled, or apparently I might have alcohol poisoning. So this is an instance where we see that, where we see that we annotated bodily reactions independent of their use to communicate an emotion, because that would have introduced an information leak. Subjective reports are, for instance, women, women, I love thee, or bad day. Descriptions of motivational tendencies are, we are going out tonight, or I think a very clear communication of emotions, sometimes I want to take your head and ram it into mirrors. Then there's an expression component. An expression of annoyance appeared on the emperor's face. Or finals tomorrow, huh? So this written-down version of this sound, huh, this is the expression component. So now I come back to this title. I come back to this title which I used, show, don't tell. So the question is now, do people actually make use more often of dimensions which are not just subjective reports than of the other components? So I show you again 2 of these spider radar plots, one for the Twitter corpus and one for the literature corpus, and now the dimensions are these different components of emotions. And so these are frequencies again. So now what we see is that the subjective component, which we would expect to be more often used perhaps on Twitter than in literature, because in literature this is actually something that should typically be avoided. So this actually seems to be the case. It's still used quite often in literature, but it's used more often on Twitter. Where people just report about their emotions, perhaps very directly. These event evaluations are used quite often in both of the different domains. So people describe events that happen to them on Twitter also to communicate an emotion. On literature, of course, events are just described, right, and you can infer an emotion out of this. The probably most noteworthy difference is in the expression component here. So of course, on Twitter, I would typically not communicate my emotion to you by telling you about my bodily reactions, right? Sorry, not bodily reaction, this is physiological, about my expression. So I probably won't write, I smile, or I cry, or something like this, and then I would probably just tell you about my emotion. But in literature, this happens quite often. So characters are described to have bodily reactions such that the reader can infer the emotion. So yeah, so this motivates of course also the use of appraisal theories to look into the connection of events and emotions. So now we put this into machine learning models very similar to the appraisal experiment. We have a text-based emotion classifier. Now here, this is a BERT classifier. We have a classifier which decides which component has been used to communicate an emotion. We put both things together. Now the model actually has access to the text because it's psychologically not that well motivated to keep this disentangled. And we use a multitask learning model now. This is a cross-stitch model. Yeah. So let's see how well this works. So the first question is, of course, how well can we predict components? And we see that across the different corpora, it's not always as easy. So it seems that some dimensions are easier on Twitter and some are easier on literature. So tech is Twitter, Riemann is literature. And there's also this one dimension here which seems to be very difficult. This is just because there are not many instances in there. Laura asks about how we collected the data. So this data already existed in the past. So the tech corpus is an established emotion corpus, and the literature corpus is a corpus which also— which we built, but it already existed. So I— the Twitter corpus has been annotated by distant supervision. They used hashtags for emotion labels. The literature corpus has been partially— has been sampled in various ways, but I don't think that there are particular exclusion or inclusion criteria. So now, does this component prediction help the emotion categorization? So I report for both corpora, Rehman literature and tech Twitter, macro and micro average. We see that the micro scores are higher. It's an unbalanced corpus. And this is just making use of the text. And the— when we, when we now include this component prediction, we see that actually not a lot happens. So this horizontal lines here, these are aligned with the micro-average scores just to be able to easily compare against the text prediction here. Yeah. So we see that including it just simply doesn't really help. Now what happens if we— the predictions of the components— what happens if we use the gold annotations of the components? And we see that this actually helps. So when we tell the model this emotion is communicated by a subjective expression, it helps the model to distinguish the different emotions. So now this is not a real-world system again because we make use of these gold annotations. Now the question is, can we make a usable system out of this? And when we put this into a multitask model, cross-stitch neural network, we see that there's actually quite some improvement for the emotion classification. So this seems to work quite okay. So now I want to answer a question, not Sure if I missed this, but are emoji considered as motor expression in the Twitter data or something else? I must say that I can't answer the question. I would need to look into the annotation guidelines. I think it would make sense if something like a particular smiley or like a disgust smiley or one with tears could be used, but not necessarily consistently, because when you have— when you cry and tear and you express tears, this would be an unconscious reaction, so it would be a bodily reaction, while— and I mean, the disgust smiley would as well, but perhaps other smileys would more be an expression component. So I must say that I don't remember exactly how we phrased this in the guidelines. I'm Sorry, I can't answer this at the moment. Alright, so let's summarize what I talked about, and also let's talk a little bit about critical aspects of this work and future work. So what we— what I would like to summarize is that we know quite a lot about emotions in psychology, and I would like to propose or motivate to look into these things when we model such things in NLP. We shouldn't ignore that just because it helps our understanding about emotion communications. It might not always help the performance, but it helps to get linguistic knowledge and computational linguistic knowledge. So I particularly looked into the emotion component process model and appraisal classification in combination with emotion classification, and we've seen that the emotion component prediction actually helps to improve the emotion classification, while the appraisal prediction has the potential to improve the emotion classification. And I would like to note that psychology has quite a lot more to offer about emotions, for instance, regarding regulation theories, how people deal with their emotions, which has not been, to my knowledge, been modeled computationally. And I think it would be very interesting. Some other questions that you might ask and which actually constitute future work is if the way how we annotate appraisals actually makes sense. So you could ask, can other people actually annotate the appraisal of somebody else based on event description? Is there sufficient information in there? And I must say that we don't really know. We don't know if the annotators agree regarding the appraisal of the original experiencer because we don't have access to the original experiencer's appraisal. So to look into this, what we are currently doing is we are building a quite large corpus. We plan for at least 12,000 event descriptions where the authors who described the event directly report their appraisal, such that we have access to the original appraisals. And then we can compare this to the appraisal annotations of annotators to see if these are aligned or if there's perhaps emotion information missing. Another question that you could ask is, for instance, in social media, is it actually sufficient to look into text? Shouldn't you also look into images? Because these are also used to communicate emotions. Is this sufficient to look into text? And currently, we're not really entirely sure. It seems that images from some preliminary studies are mostly redundant, but not always. So this is something that we are currently working on to look a little bit closer into this. And when you look into examples on Twitter, you can see that images play various roles for emotion communication together with text. So in this image here, for instance, the author uses this image in an illustrative manner. They want to communicate sadness and they use this, this picture from Lisa Simpson in a dark environment on a bed. This here is an entirely different use case. This is a picture which has been shared in various countries. In, and unfortunately also in Germany, where the person says Germany in 2030 will look like this, and they try to communicate with this that blonde people are in the minority, and apparently the audience of the person who tweeted this is afraid of people not being blonde or something like this. I actually must say that I don't fully understand this, but you can only interpret this tweet by putting together the text with the image. And another thing is this one here, which is about a conflict in some area, and you can only interpret how bad it is actually when you can see that there's a tank actually depicted here. So it seems that there are some cases where images are actually necessary to, to look at. And finally, you could also say— I didn't talk about agreement scores in this, in this presentation, but typically when you do emotion annotation, the internal heterogeneity agreement is pretty low. I mean, it's not like 0.2 or 0.3, but it's difficult to get something that's higher than 0.6 or 0.7. And you could claim, okay, perhaps this task doesn't really make sense, emotions are terribly subjective, could you actually reach high agreement and therefore develop reliable systems? And I would say that they are clearly subjective and but we hope to reach high agreement. We just don't know yet exactly how. So what the research direction that I propose here is to aim at a better understanding what causes disagreements between annotators, and that's also something that we currently study. So that's all I wanted to say. For the case that you're interested in emotion analysis, I would also like to advertise our class on emotion analysis, which you can find with this link. I also make the tech code, the tech slide code available if you want to teach emotion analysis yourself in your university. I'm very happy to support you with material, so just send me a mail and I'll give you the password to the tech code. The videos and slides are just freely available and also exercises. So thank you very much. Thanks again for the invitation. It's a pleasure for me to to give this talk here in this environment. Thanks to quite a large set of collaborators. Thanks to you for your attendance. And I'll be happy to answer questions now. And some of them have already been in the chat. Thank you.