2023.cl-1.1
The paper investigates the use of psychological appraisal theories as a structured framework for emotion analysis in text. The authors introduce crowd-enVENT, a newly created English corpus of ~6,600 event descriptions annotated with 12 emotion categories, a "no emotion" class, and 21 appraisal dimensions, collected via a two‑phase crowdsourcing pipeline (generation and validation). They detail the theoretical background of appraisal models, the design of the annotation schema (including author vs. reader perspectives), and report extensive reliability analyses showing moderate to high inter‑annotator agreement and limited influence of annotator traits. Multiple modeling experiments are conducted: (i) predicting emotions directly from text (T→E), (ii) predicting appraisal dimensions from text (T→A), (iii) using predicted or gold appraisals to improve emotion classification (A→E). Results demonstrate that (a) text contains sufficient cues for both humans and classifiers to recover appraisal ratings, (b) appraisal prediction is feasible with F1 scores around .75 on average, and (c) incorporating appraisal information can correct misclassifications of emotion models, especially for low‑agreement cases. The authors discuss the implications of treating appraisals as latent variables for emotion modeling, outline limitations, propose future research directions (e.g., personalized appraisal predictors, cross‑lingual extensions), and address ethical considerations regarding privacy and potential misuse of emotion detection technologies. The corpus, annotation tools, and pretrained models are released publicly.
Topics
Motivation and Theoretical Foundations for Appraisal‑Based Emotion Modeling
Explains why traditional affective computing approaches (basic emotion or dimensional models) are insufficient and how psychological appraisal theories provide a cognitively grounded, fine‑grained framework. Summarizes the 21 appraisal dimensions derived from seminal works (Scherer, Smith & Ellsworth, etc.) and their theoretical links to the 12 discrete emotion categories plus a “no emotion” class.
crowd‑enVENT Corpus Construction and Annotation Schema
Describes the two‑phase crowdsourcing pipeline that yielded ~6,600 event descriptions. The generation phase collected author‑perspective texts and appraisal ratings; the validation phase re‑annotated a subset from the reader’s perspective and gathered demographic and personality data. Details the dual‑perspective labeling (author vs. reader), the 12 emotion categories plus a “no emotion” class, the 21 appraisal dimensions, and reports inter‑annotator agreement metrics (Cohen’s κ, RMSE) showing moderate‑to‑high reliability and limited influence of annotator traits.
Modeling Approaches: Text‑to‑Emotion, Text‑to‑Appraisal, and Appraisal‑to‑Emotion Integration
Outlines the neural architectures used for (i) predicting emotions directly from text (T→E), (ii) predicting appraisal dimensions from text (T→A), and (iii) feeding gold or predicted appraisal vectors into emotion classifiers (A→E). Discusses training regimes, loss functions, and how appraisal information is incorporated to improve emotion classification, especially for low‑agreement emotion classes.
Appraisal Prediction Performance
Presents the empirical results for predicting each of the 21 appraisal dimensions from text, reporting per‑dimension F1 scores and RMSE (average F1 ≈ .75). Highlights the strongest and weakest dimensions and analyses factors influencing prediction quality.
Emotion Classification Gains and Error Correction via Appraisals
Shows how appraisal‑aware models outperform text‑only baselines, providing macro‑average F1 improvements and detailed gains for specific emotion categories. Includes concrete examples where incorporating (gold or predicted) appraisals corrects misclassifications made by standard emotion classifiers.
Contributions, Limitations, Future Directions, and Ethical Considerations
Synthesizes the main contributions (new corpus, annotation tools, pretrained models, empirical validation of appraisal as latent variables). Reflects on the validity of appraisal theory for NLP, acknowledges limitations (e.g., single‑language, event‑centric focus), and outlines future work such as personalized appraisal predictors, cross‑lingual extensions, and richer contextual modeling. Discusses ethical safeguards concerning privacy and potential misuse of emotion detection technologies.