The paper introduces "jury learning," a novel supervised learning framework and interactive system that models individual annotators, composes synthetic juries from selected groups, and aggregates their predictions using a median‑of‑means approach. By explicitly representing annotator heterogeneity, the system surfaces dissenting opinions and lets practitioners choose which voices a classifier should emulate. The technical implementation combines BERT‑based content embeddings with a Deep & Cross Network (DCN) to predict each annotator's label, and then aggregates juror predictions to produce a final decision. The authors evaluate the method on a large toxicity detection dataset, showing that practitioner‑constructed juries altered 14% of classifications and improved metrics such as recall for minority groups. The paper also presents an interactive UI for juror selection, visualizations of annotator agreement, and discusses broader implications for algorithmic fairness, transparency, and value‑sensitive design. Related work spans crowdsourcing, label distribution learning, and value‑sensitive algorithm design, situating jury learning as a bridge between human‑in‑the‑loop ML and responsible AI.
Topics
Jury Learning Architecture: Modeling Annotators with BERT and Deep & Cross Network
A supervised learning framework that trains a separate prediction model for each annotator using BERT‑derived content embeddings combined with a Deep & Cross Network (DCN). This per‑annotator modeling captures heterogeneity in labeling behavior, enabling downstream aggregation of diverse viewpoints.