The paper introduces jury learning, a supervised‑learning framework that explicitly models individual annotators in a dataset and lets practitioners compose a "jury"—a weighted set of annotators or demographic groups—to determine a classifier’s prediction. By training a deep architecture that combines a pre‑trained language model (e.g., BERTweet) with a Deep & Cross Network recommender system, the model predicts each annotator’s label for unseen inputs, then aggregates sampled jurors using a median‑of‑means estimator. The authors provide an interactive interface for specifying jury composition, visualizing juror predictions, exploring counterfactual juries, and resampling to assess uncertainty. Applied to a large toxicity‑detection dataset (107 k comments, 17 k annotators), jury learning achieves substantially lower mean absolute error (MAE) for individual annotator prediction (0.61 vs 0.90 baseline) and for aggregate jury verdicts (0.27 vs 0.41 baseline). A user study with 18 content moderators shows that participants construct juries with far greater demographic diversity (2.9× more non‑White, 31.5× more non‑binary jurors) and that these juries flip 14 % of classification outcomes, especially on highly divisive comments. The paper discusses design implications, ethical considerations (procedural justice, privacy, potential misuse), limitations, and future directions, arguing that jury learning offers a principled, interactive way to make explicit whose voices a model emulates. The work is situated within a broad interdisciplinary literature, citing studies on spam bot detection, label distribution learning, deep & cross networks for ranking, e‑commerce recommendation systems, designer‑in‑the‑loop algorithmic trade‑off communication, online community governance, emotion perception variability, fake news detection surveys, and value‑sensitive algorithm design, underscoring the relevance of annotator modeling and diverse stakeholder perspectives across domains.
Topics
Jury Learning Framework: Modeling Annotators as a Weighted Jury
The core concept treats each annotator as a potential juror and lets practitioners compose a jury—by individual IDs or demographic attributes—whose weighted aggregation determines the model’s prediction. This makes explicit whose voices the system emulates.
Per‑Annotator Prediction Architecture: BERTweet Integrated with Deep & Cross Network
A deep model that fine‑tunes the BERTweet language encoder and couples it with a Deep & Cross Network recommender‑style component to predict each annotator’s label for unseen comments, providing the per‑juror scores that are later aggregated.
Interactive Jury Composition Interface
A web‑based tool that enables users to select jurors by ID or demographic group, visualizes individual juror predictions, computes median‑of‑means verdicts, and generates counterfactual juries via quadratic programming to explore minimal changes that flip outcomes.
Empirical Evaluation on Large‑Scale Toxicity Dataset
Experiments on 107 k comments with 17 k annotators show lower MAE for per‑annotator prediction (0.61 vs 0.90) and for jury verdicts (0.27 vs 0.41). A user study with 18 moderators reveals that constructed juries are far more demographically diverse and flip 14 % of classifications, especially on highly divisive comments.
Design Implications, Ethical Reflections, Limitations, and Future Directions
Discussion of procedural justice, annotator privacy, potential misuse, and ecological fallacy, together with current constraints (single domain, reliance on explicit demographics) and proposed extensions such as richer identity representations and unsupervised voice discovery.
Interdisciplinary Foundations and Related Work
Survey of prior research—including spam bot detection, label distribution learning, Deep & Cross Networks for ranking, e‑commerce recommendation, designer‑in‑the‑loop trade‑off communication, online community governance, emotion perception variability, fake‑news detection surveys, and value‑sensitive design—situating jury learning within a broad stakeholder‑centric ML literature.