Roman Klinger – Emotion Modelling in Text and Social Media: The Contribution of Psychological Models

Kind introduction. The standard question in online lectures: can you hear me? Can you see my screen? Okay, that's perfect. If this changes in between, please let me know. Okay, so thank you very much. Yeah, and I see thumbs up. That's great. So how I would like to handle this is, if there are questions in between, for small questions where something is unclear, please feel free to use the chat. I can see the chat. And I'd be happy to answer small questions in between. And yeah, longer questions perhaps we'll do afterwards. So yeah, thank you very much for the, for the invitation. I'm happy to be in Graz. I would be even more happy if I'm actually in Graz and could have discussions face-to-face, but this is nearly as good. Yeah, so today I would like to talk about emotion modeling. So it's not much about resources, though I also talk about resources. And I will talk about modeling emotions in language, particularly while we look into what psychological models can help us— how psychological models can help us. You introduced me already, so I'll keep this part short. I am from Stuttgart. It's a medium-sized state university. With a focus on technical topics, but also on cultural sciences and has a real interdisciplinary focus. In there, I am part of the Institute for Natural Language Processing. This is quite a large institute for such a specialized topic, I would say. And this institute covers more or less all research areas in the field of computational linguistics. We also have our own bachelor and master programs in computational linguistics. where the master's in English, and we contribute to other programs, for instance, digital humanities, computer science, data science, and also some others. Inside of the Institute for Natural Language Processing, I am part of the chair for theoretical computational linguistics. This is what these days group pictures look like. And in, in, as part of that, I work together with a couple of PhD students and projects. members, and we do research across a set of different, different areas, mostly around emotional analysis and information extraction. So for instance, with Laura and Enrica, we focus on emotion modeling. With Aswathi and Jarek, we work on generation of emotional instances, and we also do applications in some other areas, for instance, in biomedical domain with Amelie, and also in the financial domain with Felix. So I'm a computer scientist by training. You might recognize this in the way how I approach things, but I also studied psychology as a minor subject. And nowadays my research foci are around machine learning, biomedical text analysis, social media mining, digital humanities, and with one focus is modeling psychological concepts and particularly emotions. The slides, for the case that you've not found them in the chat yet, and perhaps you came a couple of minutes late, The slides are online already. You find them with this link, and if you're fast, take your smartphone and scan the QR code if this is better for you than typing the link. And the work that I present now is mostly a collaboration with these people: my former PhD student Yevgeny Kim, my current PhD students Laura Oberländer and Enrica Troiano, and a couple of other students and collaborators who are listed here. So let's get started. I'll first talk a little bit about emotions and emotion analysis and then focus on a set of more specialized topics. Most of our research is organized like that. We are interested in the content of some message. This message comes from somebody and is sent to somebody, and we are interested in just looking into the content of this message and just By looking into the content of this message, we would like to understand what is the content, what's the content itself, what are properties of the sender. So for instance, what was the emotional state while writing the message? What might be the impact on the receiver? For instance, how might the emotional state of the receiver change when the message is received? And also the impact of the context and the relation between sender and receiver. And of course, today we talk about emotions. This follows typically a particular research lifecycle. We come up with some hypothesis, research hypothesis, and this is typically phrased somehow in a way like this: how is some particular kind of information realized in data, mostly language? And when, when we have some hypothesis about this, our goal is to create a useful system, and this can then be applied to different fields, for instance, digital humanities, literary studies, etc. Why do I call this a research lifecycle? There's no cycle in there yet. Well, we, we need to adapt this hypothesis every now and then depending on our findings in these different steps here. So it might be that we see that our system that should be useful is actually not useful, and that might be the case because our hypothesis about how information is realized in— how emotions, for instance, are realized in language is wrong. And then we need to adapt our hypothesis and do another research cycle. And today we'll talk about emotions. So to get started, let's look at a couple of examples. So let's look at some sentences and ask ourselves, and for these sentences, which emotion is associated with this example? And also ask yourself, or let's talk about that, how can we recognize that this emotion is expressed there. So if you want to express yourself regarding these questions, I think that you can't turn on your microphone, so just use the chat, please. Okay, so what about this sentence here? She became angry. So what is the— which emotion is associated to this? Anger. Yes, so that was probably pretty easy. How did you recognize this? Yeah, so there's the word angry in there. So the emotion is directly expressed in this sentence. So thank you for answering this. I know this was not too challenging. Yeah. But, but this is helpful to understand that emotions can be communicated by directly expressing them. But what about this one? A tear was running down his face. Which emotion? Sadness. Many people say sadness, and actually more people say sadness than before. And now some people also say joy, because that might also be associated to this, right? So, so some people also express that they are not really sure. So frustration. Yeah, so there's a tendency to negative emotions. Grief, which would perhaps fit into sadness. And yeah, Ezra, you're right, context is lacking. I'm sorry for that, but I'll try to keep text examples short. So how did you recognize this? One person already said because this is expressed with the word tear. So perhaps a tear itself is not sufficient to know which emotion is expressed. Yeah, but when you see that somebody's saying a tear is running down the face, then you have an association with that. So what we do here is we express— we interpret a bodily reaction, actually. And we know that this bodily reaction is typically associated with an emotion. So most of you had the first association that it's about sadness, that the physiological reaction of a— of the generation of a tear is associated to sadness. Some of you also came up with other interpretations, and some of you say that the actual combination of tear and face is important. We will go for a walk. Is this associated to any emotion? Anticipation? Not necessarily. No, no emotion. Not really. Not necessarily. Okay, yeah, I agree, not necessarily. But if you think that this is written by a person who's, who's telling somebody else, we will go for a walk, then it might be that you interpret this as a motivation to do something which might lead to a particular emotion. So I'd like to argue that it's possible to interpret this sentence as the description of a motivational tendency to do something which might be part of an emotion Which is perhaps positive. Yeah, so but obviously it's less clear here, and it seems that we don't really agree. When somebody says this, what was that? Could this person feel an emotion? Surprise, perhaps. Yeah, fear. Yeah, so the set of emotions which you're typing is now a bit. Greater. How did you come up with that? So why are you saying surprise, anger, doubt, disappointment, irritation? How did you recognize that? Earlier experiences. Okay, past inflection. So now you're, you're pointing already to language properties. What? Abruptness. Yeah. So, intonation, yes, okay, so we— this was not intentional. So what I'd like to argue is that here a person doesn't know something. Yeah, there's lack of knowledge, perhaps uncertainty, and when you're uncertain about something, this can also enable a particular set of emotions. And I think, I mean, you didn't express this like that, but my impression is that you agree with me. What about this one? Sometimes I want to run away. Which emotion? Frustration, sadness. Yeah, and this is because of a, of an expression of a motivational tendency. Last example: he raised his hand. And while I wrote this, I had such a gesture in mind, something like this. Yeah, so he raised his hand. Anger, reaction. Yeah. Anything else? Frustration. Okay, so intention to speak. Yeah, that would not be really an emotion. So what I had in mind is that this person might want to hit somebody. So it could be that you want to interpret such an expression as something that might cause fear or perhaps expresses anger, and that would be A, a, an expression then. So what I wanted to discuss here— it's totally fine if we don't agree, yeah— what I wanted to bring up here as a motivational example is that we need to think about a couple of things. One is, what is the appropriate set of emotions? You came up with a lot of different emotion names in the chat, yeah. And also how they are expressed and how can they be recognized. And what we will briefly talk about sets of emotions now and then talk about how we can perhaps recognize them. Thanks for the active participation. So how can we define a categorical system of emotions? Which of these emotions which you put into the chat should we use and which perhaps not? And this is something where we can already look into psychological theories. And one theory that received a lot of attention in the natural language processing community and also beyond, of course, is the theory by Ekman, a theory of basic emotions. And what he did is he defined properties, characteristics of emotions which would make them basic fundamental emotions. I'm not going into all of those. This is the set of properties an emotion should have such that you could call them a fundamental or basic emotion. And there are some things in there that really caught a lot of attention, particularly this distinctive physiology. So this is something that I would even say this theory is sometimes reduced to. And when you search for Ekman emotion theories, what you often find is such depictions. And you find many of those. You could also find this one if you Google for it. And what you see here is that one idea is that emotions can be discriminated, can be distinguished by expressions in the face. But there are also a couple of other things in there. For instance, that emotions are associated to events, or that there is an appraisal, an evaluation of something, or perhaps of an event associated to this. So this already serves us with a set of emotions which seem to be interesting, important, but they are just separated from each other. So we have a discrete set of emotions, but no relation between them. We don't know which of them are similar or not. There are other theories of emotions which try to tackle this issue or leverage this issue. So are these categories, for instance, somehow structured? And one theory that received a lot of attention, particularly in the natural language processing community, is Plutchik's wheel of emotions. This is also following fundamental emotions, but Plutchik puts those into relation to each other. So for instance, he says joy— you can see my pointer, right? So joy is an opposite of sadness, and fear is opposing to anger. And there are also mixtures. So for instance, love is between joy and trust. So there are relations between these emotions. This is highly debated in psychology if laugh isn't perhaps a fundamental emotion, or if it's really a complex emotion. But in natural language processing, this is, this, this is used as a set of emotions pretty often. One thing that you might find surprising in there is that the property which makes emotions opposing to each other is not the same between all emotion pairs. So between joy and sadness, It seems that what makes those an opposing pair is different from anger and fear, which are obviously both negative emotions. So perhaps this is not the right model to, to actually characterize emotions. And another model that I would like to mention is one which is actually a model of affect. So perhaps we should not just put emotions into these predefined categories. Perhaps we should characterize them in a different way. And yeah, the model of affect is also one which received a lot of attention in natural language processing, where we only have 2 variables, but these are continuous variables. The variable of valency, how positive is something, and arousal, how activated are you. And if you, if you just use these continuous variables, you can put discrete categories into this 2-dimensional space and measure similarities of those. This is not really a model of emotions. It's a model of affect, which is more fundamental process in the body, I would say. So now I have shown you 3 different emotion models which are pretty fundamental, and all of them received a lot of attention in natural language processing. So let's go a little bit away from emotion models and look at what we want to do. So what we want to do is we want to take a text, so for instance, a tweet by somebody who wrote this on Twitter, We would like to put this into an emotional analysis system. And now I define the goal. We would like to get a categorical estimate, for instance, that this tweet here is associated with the emotion joy. This is what we want to do. So this is interesting from a couple of different perspectives. It's interesting from computational— for computational linguistics and modeling, because we need to somehow understand how emotions are realized in language, and we need to computationally model that. It's also interesting for psychology and social sciences, I hope, because we might learn more about emotions and what they cause. And there are many applications ranging from humanities over social sciences to pharmacovigilance, robotics, intelligence agents, and many, many more. So this is an event about literary studies, so I'll show examples from literature. So when we— if we have such a system where we can put text in and get an emotion out, What we can, for instance, do is— and this is something that Reagan and colleagues did in 2016— they can take a book, separate this into different chunks, and measure then the emotion. What they did is they only measured the amount of happiness in each of these chunks, in each of these parts, and then you can get a curve out of this. You can measure how the emotion changes over, let's say, time. In a book. And what these authors did is with some emotional analysis system, they analyzed a lot of different books and found that there are fundamental patterns in this book. So for instance, you could see that there are books which start with low happiness and then go up. Other books start with quite some high amount of happiness and then go down and then go up again, etc. And then they also mention which books are prototypical for that. One question by Dominic is how this chunking is done. I'm not entirely sure. I think that Ragan et al. did a sliding window actually where they start on some point and they move this window through the text. This is nice because then you can also— you get some smoothing property also through the text. I'm not entirely sure. It might be that they also— that they just split this into discrete chunks. I'm not entirely sure about this. We also did something like this. Reagan et al. only looked at happiness. With Yevgeny Kim, we did this a couple of years ago with a set of different emotions where we wanted to know also if the development of emotions— this is the set of emotions we looked at— and different genres are somehow related. And it seems that there are some interesting patterns in there. For instance, that adventure stories have a pretty high peak of trust at the end. Yeah, everything's good at the end, similarly to science fiction. So now I was talking about the, the black curve, which is an average. The blue curves in there are prototypical examples that are very similar to the average. And there we could, for instance, see that sci-fi stories have sometimes a peak in the middle for anticipation and also for surprise. There are some other patterns in there, but actually to really get something out of this, you need to look into individual stories and not only on averages. We were not the first doing this with multiple emotions. Saif Mohamed did this for Shakespeare, for Hamlet and As You Like It. And he looked at joy, trust, and fear, and how these change in stories and how they are distributed in stories. And of course, you can also do this in other genres, not only literature. For instance, Stotz in 2019 measured the valence of song texts and how they changed over time. I think what we actually see is that not a lot changed over time. Metal and industrial music has always been negative and probably will always be. And while pop music and gospel and soul is typically pretty positive, and hopefully stays like that. Yeah. I think there are some developments. For instance, punk music had some ups and downs here. Yeah. And the same authors also looked into speeches by presidents. So the technology is essentially the same. So these should be motivational examples why we want to have emotion analysis systems which work well. As a reminder, the research I'm interested in mostly looks like this. We want to know which emotion is expressed in text. And we also want to know towards whom and what caused the emotion. We'll see in a minute how we can do that. And we want to know what is the emotion that's expressed by somebody who's writing a message, and what's the caused emotion in the recipient. Not all of these questions always make sense. So if you see the author of a book as the sender of a message, you're probably not so interested what the emotion was while writing the book. Yeah, but you might be interested in what a character in a text, in a novel for instance, expresses and what that might cause. So not all of these elements make always sense. So what I did not tell you yet is why these psychological models that I introduced, why, how they might be helpful for natural language processing, and this is what we will look at more Yeah, now on this slide, but also in another part of this talk. So for this Ekman model, I'd say that this mostly influenced natural language processing in the way that it serves as a categorical system for text classification. And that is essentially it in natural language processing. It's used— it's been used much more in image analysis, for instance, for facial recognition, etc. Plutchik, the same thing. However, there is also quite a lot of work which builds classifiers which make use of these structures. So for instance, the paper which I'm citing here by Suttles and Eyd, they built a set of different classifiers which distinguish, for instance, the opposing pairs of emotions and show that this can be helpful to predict emotions. Russell's model also serves as a basis for regression tasks, this valence-arousal model. Today, we'll mostly rely on these models as sources for categories. Another thing that I would like to argue is that emotions are highly linked to events, and this is something where we'll look into other emotion models in a couple of slides. So emotions and events are linked in at least 2 ways. One is that emotions are events. And the other is that events cause emotions, and both are things that I would like to talk about now in a little bit more detail in the next 2 parts of this talk, this one and this one. So emotions are events. Let's say we have a textual expression like this: Donald is happy about his birthday present. So that's good for Donald, right? So this is clearly an event. And when you look into frame semantics and the database FrameNet, which structures sentences based on frame semantics, then you'd find that FrameNet has an emotion-directed frame which would annotate the sentence like this: Donald is the experiencer of a— of an emotion, happy is a word that triggers this frame, and there's a stimulus in here which is his birthday present. Present. This is what makes Donald happy. This also motivated the task of emotion semantic role labeling, and I'll talk about this starting in the next slide. On the other side, events cause emotions. So for instance, if we have a textual expression like this, there is a car on fire, somebody might see that and think this is a relevant event. So if an event is relevant, the person might develop an emotion. So for instance, here, the speaker who says this or writes this down might develop the emotion of fear. If we want to be able to interpret events like that regarding the emotional connotation, we need to be able to understand events. There's not so much work on this. There are 2 papers, these 2 here, which build rules to analyze expressions regarding the event structure and then infer in a rule-based manner which emotion is expressed. Alexandra Balahour built a database of events and how they are associated to emotions. And I'll talk a little bit more about this afterwards. So now I would like to, to focus on the aspect that emotions are events. Everything's clear so far, or are there questions already? Okay, so emotions are events. Thumbs up. Thank you very much for the thumbs up. That's, that's actually a really motivating emoji. So let's look at this sentence here. When I mentioned the house, he seemed surprised. So if we annotate this in the way how I introduced it on the slide before, We could say that he refers to a character. We would need to do coreference resolution to know who this is. But let's say this is a character. There's also an event here, the mentioning of a house. And there's also an emotion word, for instance, surprised, which expresses surprise here. And now we could annotate relations in there and say, or express in an annotation, that surprise, this emotion event surprise, is caused by this event, and that this emotion is experienced by this character. So now we have a relational structure, which is a structure of this event, of this emotion event, which is expressed in this text. This looks pretty simple. This can easily become complicated. For instance, with such a text: All laughed at the mistake, and none louder than the 4th member of the parliament. Here, our annotators I didn't say yet that this is a corpus which we created, but I mean, we did. So here our annotator said that there is one emotion expression here for nun louder, another emotion expression, which is this one here, laughed, which could already be debated. Both point to the same target and both have 2 separate experiencers. So the interesting thing is that with such an annotation, we do not only get the information which emotion expressed, but, but we also get the information by whom this emotion is felt, which can of course be helpful for analysis tasks. And we built this corpus in a paper a couple of years ago, which Yevgeny liked to call Who Feels What and Why, which I think is a nice name for a paper. So when we created this, we of course also made this corpus available. We came up with some XML formats such that this is easily processable. We even published the annotations by all the different annotators such that researchers could also afterwards look into where the annotators did disagree, and there's quite a lot of disagreement in there. Why is this interesting? So we're not there yet, but what we want to achieve with such research is that we are able to do automatic network analysis. So we want to extract such relations automatically to be able to do network analysis. So, for instance, we want to be able to find such network structures automatically in text. This works pretty okay already. Probably many of you recognize which text this is. Yeah, the Sorrows of Young Werther. We have 3 main characters and they are in a very clear emotional relationship. I must say, I just, I just added anger here. I just guessed that Werther is often angry on Albert. I know that this is actually a more mixed relationship between those, but they clearly both love Lotte, right? And one motivational aspect why we wanted to do this is to see, to build automatic systems which are able to find these triangle relationships also in other texts, which would be a hypothesis which would generate hypotheses which other texts could be characterized as Wertheriade, as books which are highly influenced by Werther. But there are many other applications like this. If we are able to build such networks automatically, we could, for instance, also extract such a network where we have a target of an emotion in the middle. I just put COVID-19 there because I think many of us are emotional about this. And you could say that there are many— you could automatically extract, for instance, by social— in social media, that there are many people who fear COVID-19 and perhaps others are sad because of COVID-19. And if you are able— if we were able to automatically extract such networks, we could look into what do such people here or such people here have in common, what makes them sad or angry. Similarly, we could look into people who feel surprised about something regarding COVID-19 and then analyze what is surprising about this. So this is a vision. We're not there yet, but this is where we want to go with this network analysis projects. Most methods— so this was about creation of resources, right? This was not about methods. When you look into methods for such fine-grained analysis tasks, you'll find that most of the methods don't look into relations at all. Most methods that are developed on such corpora look like this. So now we have a different example. The text is John hates cars because they pollute the environment. So here we could just annotate that John is the experiencer of an emotion. There's an emotion cue, there's a target cars, and there's a stimulus pollute the environment. And if you phrase it like that, then it's a comparably simple sequence labeling task. We don't have any relation in there at all. So if there are multiple emotions in there and perhaps multiple experiences, we wouldn't get the information who's feeling which emotion. It's a simplification, but it also simplifies the methods. And this is how most of these structured analysis tasks in the emotion analysis fields are phrased or framed. And when you look at corpora which are annotated like that, then you do not only find the relation corpus which we created, but actually a set of other corpora. So for instance, there's a tweet corpus already from 2014 by Saif Mohamed, annotated cue, stimulus, experiencer, and target. There's a blocks corpus by Ghazi et al. in 2015 where they annotated stimuli. There is the corpus which I just briefly presented by Yevgeny and me. There's a corpus annotated with stimuli by Gao in 2017, and we— Laura and Yevgeny and I— we recently published a corpus with news headlines with all these different roles. All of these corpora are English except for this one. This is in Mandarin Chinese. And there are a few other corpora in other languages which I did not list here, so this is not a complete list of corpora. I think it's pretty complete for English, but it's not for other languages. Yeah, and when you look into methods, how to phrase the task of this, of recognizing these roles, what you'll find is that most of these works are either— these methods are either phrased as token sequence labeling or clause classification. If you phrase this as token sequence labeling, and now I'm only focusing on stimuli because stimuli received most attention, when you phrase this as token sequence labeling, then the task is a token classification task where you want to classify that the word pollute is the beginning of a stimulus, B, the, and environment are intermediate tokens of a stimulus, yeah, and everything else is outside of a stimulus. This is so-called IOB alphabet. Another formulation of this task could look like that. If you know where a clause is in the sentence, then you could just classify, is this— does this clause contain a stimulus? You need to recognize clauses first, but after that, what remains is the task of a comparably simple text classification. Is it a stimulus or not? We looked into this clause classification. It's actually the standard formulation in Mandarin, and it seems to be the appropriate formulation for Mandarin. This received a lot of attention, dozens of papers, perhaps hundreds of papers already in the last 3 years to do that, and they are all limiting themselves to English— I'm sorry, to Mandarin. We were interested in this, if this is perhaps also the right formulation for English. So we looked into this, and I'm not going into experimental details, but we found that this is not the right formulation for English. In English, one should phrase this task, frame this task as token sequence labeling, because the structure of the language doesn't follow this. So Emotion stimuli are typically not expressed just in a way that they— that this is limited to clauses. Yeah. So this was it about role labeling, and now I would like to move to the, to the last major part of this presentation, which is about the aspect that events trigger emotions. Everything's clear? Any, any questions in between? Yeah, so there's a comment or a question which essentially asks if such networks— I assume that you mean these network analysis which I've shown— would be able to track changes in emotions over time, for instance, in a text or in social media. I hope that I rephrase this question appropriately. And I mean, this is a discussion piece. We could, we could debate that, but this is actually the motivation for this work. If you do a time analysis, for instance, in a novel— I know that time is a difficult concept there, yeah, but, but let's say from the beginning of a novel to the end corresponds to time— or in social media over time, then you just aggregate across— over all emotions. Which are expressed independent of the group or prototypical experiencer of this emotion. So this is actually the motivation to analyze who is the feeler of a particular emotion, to be able to develop time series which are bound to particular people or to particular stimuli, which could— where the emotion experience could also change over time. I'm not aware that anybody did research in this direction. So did try to build time series in joint models of experience expression and emotion analysis. But I mean, that I've not seen it doesn't mean that it doesn't exist, but I'm not aware of such work. So there's definitely not too much work done in this direction. I think this is an interesting active research area. Are you happy with this comment? Yes. Seems so. All right, so let's move on to events that trigger emotions. And for that, I would like to introduce another emotion theory by Shera, 2005, who defines emotions as follows. He says emotions are an episode of interrelated synchronized changes in the states of 5 organismic subsystems in response to the evaluation of a stimulus event. So we have a couple of things in there. We have a stimulus. I introduced this already a little bit because I said there are stimuli which you can annotate. So stimulus events are interesting. We need to evaluate them somehow, and it's a comparably short thing. Yeah, we have just an episode. If it's not just an episode, it's probably more a mood and not an emotion. And we have these 5 organismic subsystems which we need to look at. So let's look again at this burning car, which might be a relevant event. So there's this burning car, and there's a person who observes, who sees this burning car. And I mean, this person just doesn't look very excited. The reason is that I can't draw hands, so that's why the person just sits like this. But the person might actually think this is an interesting and relevant event. So while this person sees that, there might be an emotion developed, and according to the theory by Scherer, these could be 5 different subsystems. These are the feeling, the subjective internal change could be seen as affect. Yeah, how positive, how activated am I? There's a cognitive appraisal. I think about this event and make up my mind about it. There's an expression. I could cry or start to cry, or I could— well, crying is probably not an expression. That's more something that's unconscious. But I could express my emotion by a bodily movement or saying something. There could be an action tendency, I want to do something, or there could be a bodily symptom that would be crying or trembling, for instance. And Scherer says an emotion consists of all of these different components. And now one hypothesis that we came up with is that when we want to analyze text, we need to consider these different components. So our theory is emotions can be communicated in text while— by making use of these different components. So for instance, I can express an emotion by using the feeling component. I feel bad. I could express that. I could also express my cognitive appraisal. This situation is not in line with my goals. I could also say something which corresponds to an expression. I could describe, or somebody could describe, an action tendency: let's run away. Or I could describe, or somebody could describe, the bodily symptom of somebody. For instance, he is trembling, he was trembling, or something like this. So I hope that you agree with me that each of these sentences is associated to an emotion, sometimes more clearly than in other cases, of course. And the hypothesis is that we should make use of this knowledge about emotions. So instead of just using an emotion analysis system, an end-to-end learning neural network which predicts an emotion, we might want to do more analysis, perhaps even semantic role labeling. That's not what I'm talking about now, but we want to look into this component model that I just introduced. And perhaps we can make use of that and perhaps even dissect the appraisal even more because there are another— there are other theories about this. And then based on this path here from the text through an emotional analysis system, making use of the components, making use of the appraisal, and then predict joy. So the question is, does it perhaps work better to make use of this knowledge instead of just going the path, just an end-to-end learning system which doesn't know about these components? I have no idea how to get rid of this pop-up, so it probably needs to stay. So the first thing that I want to look at is this component model here. So can we make use of this component model for emotion prediction? And this is work that we submitted to ACL. It's currently under review, and hopefully you'll be able to read this paper soon. So does this work? We annotated 2 corpora, 2 existing emotion corpora, a tweets corpus and this relational corpus with emotions of literature. We annotated this for components. The emotion labels for this Twitter corpus were the Ekman emotions. For this literature corpus, they were a set of the richer set of emotions. And we had roughly 2,000 tweets and 1,000 sentence triples from Project Gutenberg. So does this work? Is this a good idea to do that? Well, let's first build Oh, let's first look at examples. So what do we have here when we look into these components? So for instance, this cognitive evaluation, this contains instances like this, and you can guess which ones come from Twitter and which ones come from literature. I can't stop. So here a person expresses the cognitive evaluation that the person does not control the situation. That's why we put this into the class of cognitive Found my old lava lamp. So if you've been a teenager in the '90s, you might have had one, as I did. So a lava lamp is clearly associated to joy. Yeah, there's no other possibility. But to be able to know that, you need to interpret this event. There could be a bodily reaction, for instance, expressed by, she did not know, she trembled. Or apparently I might have alcohol poisoning. Stupid girl. So you might guess which one is from social media. This also illustrates that we did— we annotated bodily reactions not only if they clearly express an emotion. That would not be really possible. Yeah, we annotated such classes in the case that we found these components. Subjective: women, women, I love thee, or Bad day. These are subjective reports of an emotion. Motivation: We are going out tonight, or sometimes I want to take your head and ram it into mirrors. And the expression— where there we have both descriptions of the expression as well as expressions in the text: An expression of annoyance appeared on the emperor's face. That's not a tweet. Or finals tomorrow, huh, that's a tweet. So there, this huh is an expression. So now, um, let's do machine learning. So we use a couple of machine learning models to, to predict emotions and components. We use neural networks. Neural network, I just put this into a black box for now, but there's BERT in there, BiLSTM, CNNs. Stuff like that. And we take the text and we directly predict the emotion. Then we have a model which is configured the same way but trained to do something else. We take the text and we predict the components, these components that there is a bodily reaction expressed, etc. And then we put these things together. We use a neural network which has access to the text, but at the same time it has access to predictions of the components such that it could benefit from the knowledge about which component is expressed. If this model works better than this model, our hypothesis that component knowledge is helpful would be verified. And then we have another model where both concepts, the emotion and the components, are learned together at the same time, and the model can learn to represent these components at the same time. The hypothesis is the same. If this works better than this fundamental model here, or perhaps also the component model, then our hypothesis would be verified. So first question, how well can we predict components? So we see the results for the Riemen corpus and the TEC corpus here with an evaluation measure which is called the F-measure, which is an average of how precise are we and how complete do we make predictions. And we see that we are not doing all components equally well. There are some which are easier than others, but that doesn't mean it's not helpful. Yeah, some of the components work pretty well. So now let's look into emotion prediction. This was for just predicting the components. So what I show here is now just the model for the text prediction. I show this with 2 different averages of the F-score across the different emotions, and I show this in blue for this Twitter data and in red for the literature data. And we see that Twitter is easier than literature. That's not too surprising, I would say. Yeah. Now let's add the information of predicted components in there. These horizontal lines are aligned at the micro-average scores, just to make the comparison a little bit easier. And what do we see? It doesn't work. Yeah, so now this could be because the components just don't— the component prediction is not good enough. So let's not use a neural network which predicts the components and then uses this in another neural network. Instead, let's just use the annotations of the components, perhaps just to have a perfect component prediction. And if we do that, we actually see that the performance is better. So we can show that the component information is helpful. Just the predicted component in such a pipeline is not good enough. This is not a usable system. We don't have component annotations in real-world data, but perhaps we could use our multitask learning setup where we just learn to predict both things at the same time. And what we see is actually that the performance is nearly as good as with the gold predictions. So when we use a multitask learning neural network, this is a so-called cross-stitch neural network. Then we see that this actually works pretty well. So when we tell the model to learn components and emotions at the same time, emotion prediction benefits. This is great. Yeah. So I already said that it might be nice to also look more into appraisals because appraisal theories are very popular in psychology. And what authors the authors Smith and Ellsworth did in the '80s is that they've shown that a comparably small set of ways to characterize the cognitive appraisal of an event, a comparably small set is sufficient to discriminate between different emotions. So we use this idea and build and convert this to a text annotation task, and we did this in a CoLink paper in 2020. So we want to annotate appraisals now. So what are these dimensions? We asked annotators to annotate if a particular event that is described was perceived by the writer in a way that the writer, the experiencer of the emotion, wanted to devote further attention to this event, was certain about what was happening, Had to expend mental or physical effort to deal with the situation. Found that the event was pleasant. Felt responsible for what was going on in this situation. Felt in control, being able to change something about the situation, or the other way around. Entirely felt dominated by the situation. Nobody was able to change the situation. So these are dimensions which you could cognitively evaluate for each event you are in. And the hypothesis is that these different dimensions can help to, to describe the actual emotion. And in psychology, this has been shown. Now the question is, does this work also for text analysis? So we needed to annotate the corpus. This corpus needed to be event-centered. So with Sebastian Pardo and Enrica Troiano, we built a corpus in 2019 which is event-centered. We asked people on crowdsourcing platform, please continue this sentence: I felt, an emotion name, when, if, or that, and then they needed to continue the sentence. This experimental setup is inspired by Scherer's Study in the ISEAR project. We got 1,000 event descriptions based on this, and they look like this. So, for instance, I felt sadness when I saw a homeless cat on the street. I felt shame when somebody commented that I was looking very untidy. I felt anger when the police did not update me on a crime. So what we see is whatever happens here behind this when. It's an event description that needs to be interpreted to understand which emotion is actually expressed. And of course, in a computational analysis task, the model does not get to see these words. That would be a bit too easy. Yeah. So now we annotated this corpus, which already existed, for appraisal dimensions. And it's interesting to just look at these annotations. So what we see here is one radar plot or spider plot for each of these different emotions, and the dimensions in this spider plot correspond to the appraisal dimensions. I just point out a couple of interesting things. For instance, in fear, most instances have not been annotated with the dimension certainty. Totally makes sense. When you're not sure what's going on, it's more likely that you develop the emotion of fear. When you know what's going on, it might not be pleasant, but you're probably not afraid of it. Another interesting dimension is sadness, is the dimension of circumstance. The situation just happened, nobody was able to change it in the emotion of sadness. When you can't change this, you develop the emotion of sadness. If you were able to, to change it, you would probably not be sad, you would probably be angry. And this is what we also see here. Anger has a very low number of instances annotated for circumstance. I pointed out nice findings in the annotations. Not all of the annotation— annotated dimensions are entirely in line with psychological findings, but many are. So now the question is, can we make use of that in a system? So we use models which are very similar to the ones that I've introduced before already for the component prediction. So we use a neural network. This is now configured in a slightly different way because it's a different study, but the motivation is the same. We use a neural network which predicts the emotion from text. We use a neural network configured the same way which predicts the appraisal dimensions from text. And then we use again somehow like a pipeline setting where we first predict the appraisal and then based on the appraisal, we predict the emotion. There's some tiny change in contrast to the model that we've seen before. Because it's so well motivated in psychology, we don't give this neural network here the ability to access the text. It should be able to directly predict the emotion from only from the appraisal. And then we also had a multitask learning model where the neural network tries to predict both at the same time. So how well can we predict these appraisal dimensions? And we see that this works pretty okay. Some dimensions are more difficult than others. It's really tough to say if this is a good result or not because nobody tried this before, but the numbers look— seem to be pretty high for such complicated concepts. So the question, of course, now is, is this useful? So what I— we only discussed this black bar here, which is the average across emotions, but I also show the individual results for the emotions. So now what we have here is we have something on this corpus. This is a different corpus than we used before. That's why the numbers are different. We have 6 DF scores to predict the emotions from text. Now let's make this pipeline. First predict appraisal and then predict the emotion. And what we see here is that this does not work. So it seems that there's an error propagation. We're not predicting the emotion, the appraisal well enough to actually build the emotion on top. Same idea, let's use the gold annotations of the appraisal instead to see if we have a problem with error propagation. If we do that, we get an increase. It's not super huge, but we get an improvement here. I think something like 8% or something like this. I don't have this in my head at the moment. But we don't have these gold annotations. In the component model, what worked is the multitask learning model to get actually something that's usable. Unfortunately, here it doesn't. We were not able to find the multitask learning architecture where the emotion prediction benefits from the appraisal prediction. However, if we build an oracle— so an oracle is something that always tells you the truth— the oracle, an oracle which tells you which of the models would be more appropriate for a concrete text instance, this pipeline or this one, if we use such an oracle, to combine these 2 models, then we actually get an improvement. This is not a real-world setting, but what we learn from this is that this appraisal-based model does complementary predictions to the purely text-based model. Not a usable system, but good news because it tells us that this research endeavor is meaningful. That's it for this part. I'll have a drink and then I'll summarize. So what did we— what did we see now? What I presented today was the first work that combines, in the second part of the presentation, emotion component process— the emotion component process model and appraisal classification. We combined that with emotion classification. I find this important work because most emotional analysis systems rely on text-in, emotion-out, end-to-end learning architectures, black boxes which don't know anything about psychology. We've also shown that emotion component prediction helps to improve the emotion classification. And let's phrase it positively, the appraisal prediction has the potential to improve. Emotion classification. So now, of course, there are questions coming up. Couldn't we make appraisals help emotion classification? That should work, right? I mean, psychology tells us so. Well, we assume that we need more data. We only had used 1,000 instances, and we are working on this. We have a project where we build much larger datasets with appraisals and emotions. It might also be that the machine learning models were not appropriately Another question that you could ask is if the appraisal annotations are actually meaningful, because annotators need to guess how somebody else perceived an event. This process of guessing what somebody else thought might not work well. Yeah, this is also something that we are working on. We, we are setting up a crowdsourcing experiment where we ask people to describe events and also report their appraisal. Something that we didn't talk about in this presentation is if this knowledge about semantic roles might also help emotion classification. Yeah, I didn't present this, but the answer is yes, it's helpful. We have a paper about this. Finally, one question is, couldn't we just put this all together? Couldn't we do semantic role labeling, appraisal, emotion component, and emotion classification? Probably this is possible. We just didn't do this. I guess this is interesting and worth doing, but we didn't do this yet. Before I end, I would like to point out that we have an emotion analysis lecture available for free online, which consists of, I think, 8 sessions from row labeling over appraisals, cognitive methods, rule-based methods, dictionary-based methods, etc. So perhaps this is useful or interesting for, for one or the other. And that's it. I would like to thank you, and I'd be more than happy to hear your questions and discuss these things with you. Thank you very much.