Emotion analysis

Thank you very much. Thanks for inviting me. I feel a bit academic now after this visionary talk, and I'm even perhaps not talking about intelligence. I'm talking about emotions, and some of you might say, well, does this actually have something to do with intelligence? Can I see my slides here as well? Is this possible? Okay, no is also an answer. So, um, Some of you might know emotions because you experienced them already. And now I'm talking about how to computationally model them. So if you're interested, these slides are already online. So if you want, you can scan this barcode. I will show this barcode also at the end of the presentation again for the case that you decide after the presentation that you would like to download the slides and you can't decide already now. So emotion analysis. I am talking about emotion analysis from text. And one huge use case where we want to do emotion analysis is social media. So let's see how difficult this is. Let's say this person here wrote a tweet, and this tweet is, so happy that America is making it possible for all of its people to be married to the ones they love. So which emotion is this person expressing here? Just shout it, please. Happy. Happy, right? So, I mean, I actually have a list of emotions which are candidates: anger, anticipation, disgust, fear, Joy, sadness, surprise, and trust. And most of you just said happiness, joy.

We don't know the context, really. It could be sarcastic.

It could be sarcastic, yeah. Um, you're right, context plays a role. Um, but if we just see this one and we don't have any more context, perhaps the prior is that this person here, who's typically very serious, um, says it's joy, right? So another example, um, let's say this person here says, I'm not angry, just aggressively disappointed. So which emotion is this from the sandwich I showed you? Anger. Anger, right? So this person says, I'm not angry, but well, um, we can't really see it's anger. So this seems to be non-trivial to say, well, there's a negation with anger, but it's still anger. Another one, the Riddling Rhymer tweets, why criticize religions if the path is not your own? Don't be pretentious and get down from your So there's an emotion in there. I think you—I hope you—you—you agree with me. Which emotion is it? Just shout. Whoa! So is there any anybody voting for happiness? No happiness. Anger? Yes. Yeah, anger is in there. Fear? Some fear. Some fear perhaps. Yeah. So there are some emotions in there, but it's more difficult. So that's the task we're caring about here. This task I talk about. So what I would like to convince you about is that emotion analysis, the thing I'm talking about, is not just a fine-grained version of sentiment analysis. It's actually something more. It's something more difficult, and it's something that brings its own challenges, and it's not been addressed a lot, especially in industry. Companies. It's a challenging task, and it has many open research and application questions. So I'm telling you a lot about emotion analysis now, but I'm particularly telling you what does not work or what's under— what's not worked on a lot. So before we talk about emotion analysis, I first start by talking about what an emotion is. Before we want to model something computationally, we need to understand what perhaps other people did already in the past Psychologists, essentially. Then I'll talk about 3 use cases. We already started with social media. Then I'm talking about a field where you might say, well, how can this generate revenue? And I agree, but it's still an interesting case. I'll talk about humanities, and then I'll talk about use case in pharma industry where we take all these methods which we developed for emotion analysis into the startup which I co-founded. So what is emotion analysis? Some of you might know sentiment analysis, and sentiment analysis is often seen as just classifying text, textual instances, phrases, into a positive statement or a neutral statement, a positive statement or a negative statement. Some people say there's also neutral or mixed statements in there, but it's mostly about positive versus negative. And then there's something that's related, subjectivity analysis. That's something actually say something about their opinion, or is it just a factual statement? If it's a factual statement, might be that we are not interested in taking the statement into account when we want to analyze opinions. And then there are different kinds of emotion analysis. One has discrete categories and another has continuous categories or predictions. So let's talk about, um, these, um, Discrete categories. So this person here tries to mimic different emotions, and these are the fundamental emotions defined by Ekman. And Ekman said in the '60s, or around the '60s, what are fundamental emotions? We can say we can see the difference by the expression in the face. So that's a fundamental emotion. So let's see if we can do that. What is this guy here expressing? Joy. Joy. I agree. What about this one here? Disgust. Disgust, yeah. And this one here? Surprise. Huh? Surprise? Okay, then if this is surprise, what's this?

Large surprise.

All right, so, so I don't think it's your fault. I think it's mine, um, because I didn't play this very well. Um, but, um, what I tried to mimic here is, um, joy, anger, disgust, Surprise. Well, I'm not so happy with sadness here. And fear, which, which you said it's surprise. I tried to make fear here. So these are 6 categories, and when we look into a face, it seems to be doable to understand what's happening. But we don't want to do it based on text. There are also other models, just to mention them shortly. There's the Plutchik's Wheel. where we have also intensities in there. So the more we go into the middle of this, it's more— the emotion is more intense. So we have joy here as well, but when we go into the middle, it's more about ecstasy. When we, um, when we have, for instance, disgust, a weak form of disgust is boredom, which might surprise the one or the other. And there are also other models which live more in a continuous space. But these are typically not used for text analysis. So these are emotion models. These are models which we use as label sets to make our predictions. Still, we don't really know what an emotion is and what purpose they serve. And there are a couple of theories, and let's just pick 3 out of them. So there's the James-Lange theory, pretty old. For artificial intelligence and computer science people, it's super old. And what they— what these people said is, we have some event, some trigger, for instance, this nice snake here. And we see that, and then something happens to us. So what happens might be that my heart starts pounding, I'm trembling, I'm sweating, just because I see this, and finally I run away. And after that, after all these things happened, I fear— I feel that I'm afraid. So it's not I am afraid and then I run away. First I have these physiological reactions and then I develop the emotion. That's this theory. Well, some people doubted that. Um, still pretty old. Um, Cannon-Bard theory says emotions are independent of physiological signals, and they also did experiments, and these experiments said There is— these are independent. There's no reliable correlation between, between these 2 things. However, a couple of years later, in 1966, Stuart Valance— and this was a famous experiment, so he got this experiment named after himself, the Valance effect— came up with an experiment. So what do we see here? We see this guy, um, he doesn't listen to music, he hears the sound of a heartbeat. And he's told, this is your heartbeat. So he can hear his own heartbeat. There's a control group, we're not talking about this. And then we show this guy pictures of women without a lot of clothes. So something happens to this person, the heartbeat can rise or go down. So I just decided to take this picture. I think in reality they were less clothes. And, and then this guy was asked afterwards, which of these women did you like? And then this was not the real heartbeat. They increased the speed of the heartbeat or lowered it, and at the end this person actually believed to like the women where the heartbeat was raised. So, and without knowing that this is an artificial heartbeat. So this tends to say there is actually this reaction of first physiological reaction, and then there comes the interpretation of the emotion. So we don't really know what's what's going on. Another thing is the so-called appraisal theories, and that's one which is actually started to be used for textual analysis. And this is coming from Shayla, a bit newer, 2005. Emotions are an episode of interrelated. Synchronized changes in my body in response to an evaluation of an external stimulus. So something happens, I evaluate it, and then something happens to my body, and that's the emotion. Only if there are major concerns for the organism. So for instance, if you see, let's say, a spider running around somewhere, you might decide, okay, I need to evaluate my This event. I've seen some of you smiling now, so this seems— well, you didn't evaluate this to be a major event which is important for your organism. And when I experimented this with— in my office, actually, some people jumped backwards who were a little bit more afraid of spiders. So perhaps some others of you see cats and think, now this is a major event, I need to do something about that. So if there's a major event, then you develop an emotion. So what we learned so far is that emotions have different components. We have a cognitive appraisal, we have bodily symptoms, we have reactions, we have expressions, we have also subjective perceptions. We didn't all agree. So now we have an idea of what emotions are. Now we move forward to textual analysis. What do we actually do with text? Which of these theories do we use for textual analysis? I assume that at the end of this presentation you agree with me that most of the previous things psychologists did research on are underexplored for textual analysis. So let's look at social media. You could say, okay, we have these 6 emotions, let's just represent them with emoticons and search for them in text. I assume that you agree with me that it's not very likely that this will find all the cases of emotion expresses— expressions, for instance, on Twitter. So we need to do something else. So I consider myself a computational linguist. So what computational linguists do is they annotate corpora. So several people annotated several corpora, and this is just a list of different corpora which are publicly available. And you can download them, look at the textual instances, and each of these textual instances, for instance a tweet or a blog post or a sentence, has a label. This is this and this emotion. And then you can learn from the data and do something with that. Now I'm talking about So one corpus here, this SSDC corpus, because this is one I built together with a couple of colleagues. So the task was here to annotate tweets. We used these nearly 5,000 tweets which were already published before in a shared task in a competition where people were invited to build systems for sentiment and stance detection. So sentiment is the thing which I'm interested in here. And the research question was, how difficult is it to actually annotate such tweets? We did an experiment at the beginning. Now let's see how difficult is it actually to annotate these tweets? How many people do actually agree to assign a specific emotion? And also, which annotation layers exist— interact? So the question is, how far away is emotion analysis from sentiment analysis? Is it just the same thing, or is there a clear correlation between them? And of course, also, how well is it possible to computationally estimate such things? So let's look at the annotated data. So what do we see here? We see the count for a label. So we will see bars here. How often, how many anger tweets we have, how many fear tweets we have. So this is the count on the y-axis. This block of bars here, which I will show in a second, here on the left, this will be the count where we accept a label if at least one annotator assigned a specific emotion. So if one person said, this tweet is about anger, then we accept this as a true annotation. For instance, this one here, we need more than 50% of the annotators to have assigned a tweet the specific label. So when we accept just one person to assign a specific label, then these numbers are pretty large. So that's fine. Now let's go down with this threshold and accept only the label when more than one person agrees. And we see that these numbers actually go down pretty dramatically. So if we only accept the label if all of the annotators— we had up to 7 annotators— agreed, we see that these numbers are much lower than if we only accept this one. And for computational linguistics or natural language processing tasks, these are pretty dramatic values. So the agreement is really low. So the question is, why is this? Let's look at some examples. Some difficult examples. So this one here: That moment when Canadians realized global warming doesn't equal a tropical vacation. So which emotion is in there? Our annotators said— more than one annotator just said surprise, sadness, and anger. And I think we can agree with that. It all makes sense. Just from different perspectives. You said in the beginning context plays a role, but it's not only context, it's also previous assumptions, it's also previous knowledge. So there's surprise in here, but the tweet is not really about a surprise. The tweet is more about that. But this is a— these are different perspectives. Another difficult example is this one: 2 pretty sisters are dancing with a cancered kid. So this is a pretty sad example, but it's It's a real-world example. We have these 2 people, and in the middle there's a kid who's seriously ill. And there are just different emotions you can develop when you see such tweets. You could say, I'm happy, I'm joyful, and I trust in the situation because these 2 women are so nice to the kid. But on the other side, you can also be afraid and sad about this whole thing. So it actually makes sense. So it's not trivial to make the distinction between all possible possibilities. Which emotion holds in such tweets. So, this is a table of all co-occurrences in the dataset. I'm not going into this. I'm just pointing out some interesting examples. And this is this one here. There is a number of tweets which is positive in sentiment. This is an annotation which was in there before. But it's about anger. That's something you would not expect. And it's a substantial number. Another is this one here, negative joy, 750. And another one is actually positive disgust. So how can something be positive and disgusting? So let's look at these examples. And this is a pretty common pattern on Twitter. Positive anger: let's take back our country, who's with me, no more Democrats. So this is about anger. I'm angry, but I'm asking you to change something. So there's also positive So you can't say if you see anger in a tweet, it's always negative. That doesn't make sense. That would not work. It's a different task. Negative joy. Global warming, global warming, global warming. Oh wait, it's summer. That happens pretty often that one set of the annotators caught the irony or sarcasm, found it, detected it, and others did not. It is positive. It is joyful, but it's also negative. Let's look at the one with positive disgust. I don't want to read this. Please do this yourself. So I think we can agree that there's something positive in there, but also something nasty. So now, this was human performance. Let's do this computationally. We first did some state-of-the-art models, a maximum entropy classifier and SVM classifier, and then we did even more state-of-the-art models, 3 deep learning models. When you look at these performances, you see that, yeah, we can do 58% F-score. Let's just look at the most right bar here, which is an average between how precise I am and how complete my annotations are. I'm not super complete, only 50% of the cases I can find. But in— from the cases which I find, 66% are correct. Same result for SVM. And when we move to more state-of-the-art models, the performance are pretty similar, but we find more of the results. That's a pretty common pattern. Deep learning methods are on average not necessarily much better in text classification, but they average out precision versus recall a bit better. So what I tried to communicate so far is that deep learning is Better than traditional models, but it is super slow. So in practice, you wouldn't apply this actually here. This is a challenging task. There are interesting instances here, and I don't have a solution for these instances which I showed you. So it's really a problem which still holds. It's unsolved. But it's on the other side enables a lot of analysis tasks in many different application areas. So if you, for instance, query for tweets, and we will see this later at the end of my talk in an example, you query for tweets. Tweets which include some specific entities which you're interested in, then you can automatically assign the emotions to these entities and make large-scale analysis. So we will look at one example later. So this was social media. That's a pretty popular topic. There are, I don't know how many million sentiment analysis companies out in the world. I don't think there are many emotion analysis companies in the world. I'm not sure that there's actually one which does text analysis in its focus. Now let's talk a little bit about humanities. So one question we had here is, can we characterize literature with the help of emotion analysis? So that's a totally different task. Literature might be more difficult, might be more challenging than tweets. I think every novel author would agree with me that tweets are more simplistic than novels. So let's see if this can work here as well. And there's been some work. So Saif Mohammed in 2011 already just counted with dictionaries how does the development of emotion words change in drama and in comedy. Yeah, Hamlet is drama and As You Like It is comedy. And you can see the flow of different emotions here. So he just used these 3: joy, trust, and fear. And then a pretty well-recognized paper in 2016 did the following. They just measured happiness. They didn't do anything else. They only measured happiness, mostly with a dictionary-based approach. And they found that you can cluster texts, novels, based on the shape of the emotion development. So there are texts which start with a lot of happiness, go down, go up, and go down again. You can group them by that. Others are, for instance, this one here. You start happy, Then not a lot of happiness, and then there's happy again. So this was the analysis in this paper. We were interested in, can we combine this? Can we relate this to more structured information, for instance genres? So what we did here is we used Project Gutenberg, which is a public source for novels which you can download, not the most recent ones but pretty modern ones. And they have labels, for instance, for adventure stories, romance stories, mystery detective stories, humorous stories, and science fiction. And we built a classifier based on emotions to detect the genre. So we used these emotion scores, these changes of emotional intensity measured with dictionaries over time. But we also used a standard bag-of-words model where each occurrence of a word is an indicator for a specific class. And also a model where we only used words which are associated with emotions. So, is there actually something in the emotion? And the models are pretty standard: random forest, deep learning models. Should I talk about that? Yes, I think I should say briefly something about the CNN. So, the input for this deep learning model is that we have a matrix where we have the emotion scores going from left to right. Each row is one emotion. So this, for instance, is the anger score. This is the fear score, and so on. And in addition, we have indicator numbers which tell us about where, in which position we currently are. Because typically CNNs are— don't take into account the position inside of a matrix. So we encode this implicitly. And the rest is pretty standard. So what happens if we do that? Can we just based on emotions predict the genre? So let's just use as a baseline the— we have the bag-of-words model. So where we just use words to find what's the genre. With random forest and the multilayer perceptron, we get something between 80 and 75 depending on the model. Now let's just use words which are associated with emotions but still emotion words. And it's roughly the same thing. So there's not a huge difference. Note that The baseline here is 55. It's not 0. Yeah? Because it looks better. And then we move on to the emotion scores. And when we do that, it looks pretty dramatic. I mean, it's not that dramatic because we started at 55. But it's not such a good score. It's much better than random. Random assignment is 3% or something like that. But it doesn't work well. So again, there seems to be challenges in emotion recognition which are unsolved. In literature, This seems to be problematic. We can of course do more analysis here. We can plot all these different curves for all the different genres. This is partially pretty interesting. I'm just picking out the one here for adventure and trust up here, where we see that adventure stories typically start with a lot of trust. You get to know the people, you need to get to know the characters. Then something dramatic happens. There's no trust anymore, and at the end everybody's happy again. So this is a typical adventure story, and you can find such patterns in such graphs. However, when you do that with a simple counting model, you do not take into account the different characters who experience emotions in the text. So for instance, if we have one character with this purple line here, which has specific emotional development, and another person who has another emotional development like this one here, These are artificial curves, then the average would just be a flat line. So we don't see anything. So something that also nobody worked on—I mean, we are working on this now at the moment—but nobody worked on so far is something like to extract emotions in a more structured way. So, for instance, what we are currently working on is we build models which can find such entities. So let's just look at one—at this one here. So the sentence is: When I mentioned the house, he seemed surprised. It's a snippet from a novel. There's a textual phrase here which says surprise, so this is pretty easy to detect which emotion is expressed. But we also detect the event which triggers the emotion. That's something that's absolutely not standard to do that. There are some works on Twitter which detect the cause for the emotion, but also only just 2 or 3 papers. And also we detect who experiences the emotion. It's also— it's something that's called aspect-level sentiment analysis when you only do positive or negative. For emotions, it's absolutely— nobody's doing that yet. And it's pretty challenging, by the way. So we built a corpus for that. If you want, download this, build models on that. This is a literature corpus. We are doing this on other domains right now. And when you do that, when you detect such things, you can, for instance, detect such social networks. So this is just a co-occurrence network. In the text The Sorrows of the Young Werther, where you have this triangle relationship. Werther loves Lotte, Lotte loves Werther, but Lotte also likes Albert quite a lot. And this— and of course there's also a relationship between Albert and Werther, but of course the relationship between Albert and Werther is not love, right? So this is something that when you just do co-occurrence analysis You would not see in such a network. But you would like to see that. You would like to characterize texts based on such networks, especially because the Thesaurus of Young Werther triggered hundreds of other books where everybody just built such triangle relationships in different environments. And these are some just co-occurrence networks of characters in texts which just look the same. The characters are different, the character names are different, but the roles of the characters are the same. So there's also this force in here. I mentioned this force annotation. This is something that is related to the appraisal theory, which I mentioned at the beginning. There's an event, and then somebody evaluates— the character evaluates this event and decides, is there something I should develop an emotion based on? And because this has never been done on a large scale, we did a shared task on that. And this is just a Well, the teaser is over, but you can read the papers about that. So this is the implicit emotion shared task where we build sentences from Twitter like, it's— we don't know what emotion this is, we leave away this emotion— when you feel like you are invisible to others. So when you feel invisible to others, that's an event you might evaluate and decide, okay, I should perhaps develop an emotion based on that. So you might be sad. Because of that. So we made a shared task out of that. The website is online. The papers will be published soon. This is a workshop which takes place in Brussels next— on Halloween, actually. So if you want to join, it would be great. So now we talked about social media, a pretty popular domain. We talked about humanities. And now I'm talking about something where we try to take this knowledge about emotion recognition into practical applications to actually generate revenue. So I'm a co-founder of the company Semalytics. So what we do is we have these visualizations. For instance, we see we have a distribution of different emotions, which is pretty interesting because disgust here on these pharma-related tweets is a huge part. And this is totally uncommon in related In contrast to other Twitter datasets, nobody talks talks on Twitter about being feeling disgust. Everybody talks about being happy or angry, but nobody feels oh this is disgusting. This is different in the pharma domain. People talk about what they think is disgusting. So for instance, these are some tweets. If you suffer suffer from psoriasis, postnatally prepare to turn into hideous. Scaly beast. So this is about anger. So there's somebody angry about their disease. That doesn't help so much because there's no drug name in there. But here it is. So grateful power has been restored and I was able to get my Remicade infusion today. So here this person expresses joy about a specific drug. It's related to hurricane events and so on, but that's not the relevant part here for a pharma company. So, And there's one tweet here, which is actually wrongly labeled from our annotators, but I still kept this in because it's interesting. This has been labeled by our annotators as sadness. So far today, my insurance decided to stop covering my Yomihira. My hip aching from the rain moving in. My foot is swollen and I want to cry. So this person is angry in a tweet which mentions a specific drug. But we do not want to extract the knowledge that somebody's sad about the drug because the person is sad about the absence of the drug. So we need to do target-level annotate analysis here to understand the emotion towards a specific entity to actually generate knowledge based on this. So this is also another plot of different biosimilars, different drugs which are related to a specific medical condition. And what we see here is we just sum up the numbers of different emotions, and what we see is that there are some drugs where people only talk about disgust. And if you are the manufacturer of such a drug, you want to know what's going on. You want to know why everybody's feeling disgust about this. Or just fear. Or only disgust and sadness. On the other side, there are some drugs here, for instance, there's also some disgust there, But there's also a little bit of happiness, but especially trust. And trust is something pharma companies want to have. And there's a difference between just a positive tweet about a drug and a tweet which says, I trust this specific medication. That's the difference. So, what are challenges? How did we start? We first took this corpus which I annotated, which I mentioned before. And we tested this on a small test set with these biosimilar data. And what we got were 3% F-score. It just didn't work. Now, the developers were tempted to say, well, the plots still look nice. And they do. But there's just nothing in there. So we need to improve the method here. So how can we improve the method? So one approach is we do reannotation of data. And that's somehow the most straightforward approach. It's pretty expensive and it takes time. You need to pay annotators to annotate substantial amounts of data. That's what we do because it's the most straightforward idea. However, another possibility would be to use transfer learning methods from existing corpora to improve them to work well on other— on our target corpus. Unfortunately, it's known that transfer learning works, but it's not really known which transfer method— transfer learning method works when you don't have annotated data to transfer to. So it's— this is not an established technology. So still, we tried this. We used this set of corpora which I mentioned before. We trained on all these different corpora which are here on the— corresponding to the columns. And we tested on all other— on the same corpora. So what you would expect is that the diagonal leads to high numbers, right? When you train on one corpus and you test on a test set which is not in the training set, that you get good results. This mostly works, but there are still some surprising cases. For instance, when you train on this corpus called Effective Text, you can get to 65%, but when you test it on another corpus, you only get 9%. So you can really do a very bad job when you just randomly select some dataset. So it's really a challenging task to find the right dataset for your problem. Even if the task is totally clear, you don't want to do emotion analysis. The domain matters a lot. So we are working here on improving transfer learning methods. So what I tried to tell you today is that emotion analysis is in a situation that it's backed well by psychology. We know a lot about emotion psychology. I think we even know more about emotions than about sentiment. However, it's just not completely transferred to natural language processing or AI technology. Some parts were taken out of these theories, but not all of them. There's much more to explore which has been done in the last 100 years, and the AI community just didn't take this and didn't use this. So, and what we've seen at the end is that models typically overfit to the text source or domain on which we train. We really have a problem there. We have really— it's really challenging to build models which generalize across different domains. So there's a huge potential in applications across different domains. So I think it would be great if more of you join in the emotion analysis business and build models for different applications here. Thank you very much. Thank you.