A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Automated fact-checking based on machine learning is a promising approach to identify false information distributed on the web.In order to achieve satisfactory performance, machine learning methods require a large corpus with reliable annotations for the different tasks in the fact-checking process.Having analyzed existing fact-checking corpora, we found that none of them meets these criteria in full.They are either too small in size, do not provide detailed annotations, or are limited to a single domain.Motivated by this gap, we present a new substantially sized mixed-domain corpus with annotations of good quality for the core fact-checking tasks: document retrieval, evidence extraction, stance detection, and claim validation.To aid future corpus construction, we describe our methodology for corpus creation and annotation, and demonstrate that it results in substantial inter-annotator agreement.As baselines for future research, we perform experiments on our corpus with a number of model architectures that reach high performance in similar problem settings.Finally, to support the development of future models, we provide a detailed error analysis for each of the tasks.Our results show that the realistic, multi-domain setting defined by our data poses new challenges for the existing models, providing opportunities for considerable improvement by future systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“3) For evidence extraction, stance detection, and claim validation we evaluate the performance of high-scoring systems from the FEVER shared task (Thorne et al., 2018b) 4 and the Fake News Challenge (Pomerleau and Rao, 2017) 5 as well as the Bidirectional Transformer model BERT (Devlin et al., 2018) on our data.”
MethodWeighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1×
“7 (Cohen, 1968), indicating a substantial agreement between the crowd workers (Artstein and Poesio, 2008).”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 2×
“, 2016) is a neural network with a relatively small number of parameters that uses decomposable attention, reaching good results on the Stanford Natural Language Inference task (Bowman et al., 2015).”
BackgroundInter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665 · cited 2×
“7 (Cohen, 1968), indicating a substantial agreement between the crowd workers (Artstein and Poesio, 2008).”
BackgroundFEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 4×
“This motivates the development of automated fact-checking pipelines (Thorne et al., 2018a; Popat et al., 2017; Hanselowski and Gurevych, 2017) consisting of several consecutive tasks.”
MethodInfluentialCer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267 · cited 1×
“USE+Attent is a model which uses the Universal Sentence Encoder (USE) (Cer et al., 2018) to extract representations for the sentences of the ETSs and the claim.”
MethodFact Checking: Task definition and dataset construction
Vlachos, A., & Riedel, S. (2014) · Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science · doi:10.3115/v1/w14-2508 · Cited by 402 · cited 1×
“PolitiFact14 Vlachos and Riedel (2014) As our analysis shows, while multiple fact-checking corpora are already available, no single existing resource provides full fact-checking sub-task coverage backed by a substantially-sized and validated dataset spanning across multiple domains.”
BackgroundSemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours
Derczynski, L., Bontcheva, K., Liakata, M., Procter, R., Wong Sak Hoi, G., & Zubiaga, A. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2006 · Cited by 323 · cited 1×
“The corpora introduced by Vla-chos and Riedel (2014); Ferreira and Vlachos (2016); Derczynski et al. (2017) are valuable for the analysis of the fact-checking problem and pro-a vide annotations for stance detection.”
Background
Cited by
A Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“Nevertheless, traditional fact-checking methodologies (Augenstein et al., 2019; Hanselowski et al., 2019; Atanasova et al., 2020) often incorporate simplified assumptions for practicality, leading to discrepancies when applied to complex real-world scenarios.”
Explainable Automated Fact-Checking for Public Health Claims
Kotonya, N., & Toni, F. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.623 · Cited by 195
COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic
Saakyan, A., Chakrabarty, T., & Muresan, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.165 · Cited by 91
“Recent work on end-to-end factchecking, including models and datasets, has advanced the field by addressing several tasks in the pipeline, but not all (Thorne et al., 2018, 2019; Hanselowski et al., 2019; Augenstein et al., 2019; Diggelmann et al., 2021; Wadden et al., 2020).”
BackgroundEvidence-based Fact-Checking of Health-related Claims
Sarrouti, M., Ben Abacha, A., Mrabet, Y., & Demner-Fushman, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.297 · Cited by 71
Metadata
- Authors
- Hanselowski, A., Stab, C., Schulz, C., Li, Z., & Gurevych, I.
- Venue
- Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
- Published
- 2019
- Pages
- 493-503
- OpenAlex
- W2988245244
- Cited by
- 84
- References
- 11
- FWCI
- 28.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking. https://app.cochat.ai/papers/10.18653/v1/k19-1046
- The paper
- Hanselowski, A., Stab, C., Schulz, C., Li, Z., & Gurevych, I. (2019). A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). https://doi.org/10.18653/v1/k19-1046
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.