2024.eacl-long.124
The paper investigates why some biomedical claims extracted from Twitter are verifiable while others are not. The authors introduce BEAR-FACT, a new dataset of 1,448 fact‑checked medical claims derived from the BEAR Twitter corpus, each annotated with subject–relation–object triples, evidence documents from PubMed, and a fact‑checking verdict (SUPPORTED, PARTIALLY SUPPORTED, REFUTED, PARTIALLY REFUTED, UNVERIFIABLE). An in‑house annotation study with two trained annotators examined how evidence is retrieved, focusing on query refinement strategies and confidence in evidence existence. Statistical analysis revealed that claim verifiability is strongly linked to entity‑relation patterns: positive relations (e.g., cause‑of) are more often supported, whereas their negated counterparts (e.g., not‑cause‑of) tend to be refuted or unverifiable. Annotators mainly refined search queries by normalizing entities (brand to active ingredient, abbreviations, informal terms). A crowdsourcing experiment compared lay participants and biomedical experts; domain expertise had limited impact on annotation reliability, though experts selected evidence more selectively. Finally, a RoBERTa‑based classifier was trained to predict claim verifiability from text alone, achieving .82 F1 for verifiable claims but only .27 F1 for unverifiable ones. The paper discusses implications for fact‑checking systems, highlights challenges with negated claims and environmental factors, and outlines future work on evidence retrieval methods, dataset expansion, and modeling approaches.
Topics
BEAR‑FACT Corpus: Construction and Core Annotation Layers
Details the creation of the BEAR‑FACT dataset, including claim extraction from the BEAR Twitter corpus, filtering steps, and the two annotation layers (fact‑checking verdicts and structured entity‑relation triples).
Research Questions Guiding the Study
Four RQs: (RQ1a) entity‑relation patterns influencing verifiability, (RQ1b) utility of medical entities as search queries, (RQ2) effect of annotator expertise and crowdsourcing, (RQ3) predictability of verifiability from claim text.
Annotation Procedure and Evidence Retrieval Workflow
Describes the in‑house annotation workflow: initial PubMed query built from claim entities, three‑minute search limit, query refinement actions, and confidence rating on evidence existence.
Corpus Statistics and Verdict Distribution
Presents quantitative overview: 1,448 claims across 572 tweets, distribution of verdicts (62.4% supported/partially supported, 30.9% unverifiable, 6.7% refuted/partially refuted), claim‑per‑tweet histogram, and co‑occurrence patterns of verdicts within tweets.
Entity‑Relation Patterns and Verifiability (RQ1a)
Analyzes how specific relations (positive vs. negative) and entity types correlate with verdicts; chi‑square tests show significant differences for most relation pairs, highlighting difficulty of negated claims.
Query Refinement Strategies and Their Effectiveness (RQ1b)
Identifies seven refinement types (generalizing, specifying, brand normalization, informal term normalization, abbreviation resolution, adding relation, other); reports frequencies (informal term normalization most common) and success rate (24.4% of verified claims required refinement).
Crowdsourcing vs. In‑House Annotation (RQ2)
Compares lay crowdworkers and biomedical experts on verdict agreement (Cohen’s κ) and evidence overlap (Jaccard); finds modest agreement, limited impact of domain expertise, and more selective evidence use by experts.
Predicting Claim Verifiability with RoBERTa (RQ3)
Describes the binary classification experiment (VERIFIABLE vs UNVERIFIABLE), training details, and results (.82 F1 for VERIFIABLE, .27 F1 for UNVERIFIABLE); error analysis links model behavior to claim specificity and entity normalization.