2022.argmining-1.18
The paper addresses the problem that biomedical fact‑checking models are trained on well‑formed, atomic claims (e.g., SCIFACT) while real‑world medical statements on social media are long, noisy, and contain multiple intertwined facts. To bridge this gap, the authors propose two entity‑based claim condensation techniques that transform a tweet into a concise claim representation using gold‑annotated biomedical entities and causal relations. The first method, condense_triple, builds a subject‑relation‑object triple by concatenating the tokens of the two entities with the relation label. The second, condense_seq, extracts the shortest token span that covers both entities, preserving the original wording between them. Experiments are conducted on the CoVERT corpus, which provides fact‑checked COVID‑19 tweets together with entity and relation annotations. The authors evaluate five MultiVerS‑based fact‑checking models (fever, feversci, scifact, covidfact, healthver) under four claim input conditions: full tweet, a claim sequence predicted by a Bi‑LSTM‑CRF model (Zuo et al., 2022), condensetriple, and condenseseq. Performance is measured by label‑only F1 on verdict prediction. Results show that (i) models perform poorly on full tweets, (ii) delimiting the claim improves all models, and (iii) entity‑based condensation yields the highest gains, with healthver achieving 62.0 F1 using condenseseq. An error analysis reveals cases where condensation helps (e.g., turning NEI into correct labels) and where it harms (e.g., loss of crucial context). The authors discuss limitations such as reliance on gold entity annotations, focus on causative relations, single dataset and architecture, and suggest future work on other relation types, noisy NER pipelines, and broader fact‑checking systems. Ethical considerations highlight the risk of mis‑representing user statements and the need for transparent, accountable deployment of automated fact‑checking.
Topics
Mismatch Between Social‑Media Medical Claims and Existing Fact‑Checking Datasets
Explains how user‑generated tweets differ from the short, well‑formed claims in datasets like SCIFACT, leading to poor transferability of current fact‑checking models.
Entity‑Based Claim Condensation Methods
Introduces two concrete techniques—condense_triple (subject‑relation‑object triple) and condense_seq (shortest span covering entities)—that use annotated biomedical entities and causal relations to create concise claim representations.
Experimental Setup on the CoVERT Corpus
Details the data (264 non‑NEI COVID‑19 tweets with gold entity/relation annotations), the fact‑checking models (MultiVerS variants), and the baseline claim‑sequence predictor used for comparison.
Impact of Claim Condensation on Verdict Prediction
Presents quantitative results showing that delimiting claims improves all models and that entity‑based condensation yields the largest F1 gains, especially for the healthver model (up to 62.0 F1).
Error Analysis, Limitations, and Future Work
Analyzes cases where condensation helps or harms predictions, acknowledges dependence on gold annotations, limited relation types, single dataset, and proposes extensions to other relations, noisy NER, and alternative fact‑checking architectures.
Ethical Considerations of Automated Claim Extraction
Discusses potential harms from mis‑representing user statements, the importance of transparency, and the need for careful evaluation before deployment.