2023.fever-1.3
The paper presents a fully automatic pipeline that extracts concise, entity‑centered claims from biomedical tweets to improve fact‑checking performance. The pipeline consists of (1) a custom spaCy NER model trained on the BEAR dataset to detect medical entities, (2) optional entity linking via ScispaCy to UMLS for terminology normalization, (3) claim candidate generation using the condense_seq method (extracting the text span between the first and second entity), (4) main‑claim detection with a RoBERTa‑based classifier fine‑tuned on causal claim data, and (5) a frozen MultiVerS fact‑checking model that predicts SUPPORT, REFUTE or NOT ENOUGH INFORMATION. Experiments on the CoVERT dataset show that (a) replacing gold‑annotated entities with automatic NER leads to some performance loss but still yields higher fact‑checking F1 scores than using the original tweets, confirming the benefit of entity‑based claim extraction; (b) normalizing entities to canonical UMLS forms does not improve, and even harms, downstream fact‑checking due to errors in the off‑the‑shelf linking module. Detailed analyses of NER (strict vs. relaxed evaluation), main‑claim detection (high precision/recall), healthver model behavior, and entity‑linking error patterns are provided. The authors conclude that improving NER and reliable linking are key future directions and caution against fully automatic deployment without human oversight.
Topics
Motivation: Bridging the Gap Between Biomedical Fact‑checking Models and Social Media Claims
Discusses the mismatch between models trained on synthetic, well‑structured claims and the noisy, colloquial nature of biomedical tweets, motivating the need for claim adaptation.
Pipeline Architecture: From Entity Detection to Fact‑checking
Details the end‑to‑end system comprising custom NER, optional UMLS entity linking, claim candidate generation (condense_seq), main‑claim detection via RoBERTArg fine‑tuning, and a frozen MultiVerS fact‑checking model.
Experiment 1: Impact of Automatic NER on Claim Extraction
Evaluates how replacing gold entities with automatically detected ones affects downstream fact‑checking, showing a modest performance drop but overall gains over using full tweets.
Experiment 2: Effect of Entity Normalization via UMLS Linking
Investigates whether normalizing entity mentions to canonical forms improves verification; finds a performance decline attributed to linking errors and incomplete coverage.
Component Analyses: NER, Main‑claim Detection, and Linking Error Patterns
Provides quantitative evaluations of NER (strict/relaxed F1), claim detection (high F1), healthver model behavior, and a qualitative error analysis of entity linking (correct, related, unrelated mappings).
Conclusions, Limitations, and Future Work
Summarizes that entity‑based claim extraction improves fact‑checking despite error propagation, highlights the need for better NER and reliable linking, and warns against fully automatic deployment without explainability.