The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
The Fact Extraction and VERification (FEVER) Shared Task
Open accessAbstract
We present the results of the first Fact Extraction and VERification (FEVER) Shared Task.The task challenged participants to classify whether human-written factoid claims could be SUPPORTED or REFUTED using evidence retrieved from Wikipedia.We received entries from 23 competing teams, 19 of which scored higher than the previously published baseline.The best performing system achieved a FEVER score of 64.21%.In this paper, we present the results of the shared task and a summary of the systems, highlighting commonalities and innovations among participating systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing fact-checking approaches either require access to the output probability distribution (which may not be available for systems such as ChatGPT) or external databases that are interfaced via separate, often complex, modules.”
From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Thorne et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An alternate approach is to leverage fact-verification approaches, where evidence is retrieved from an external database to assess the veracity of a claim (Thorne et al., 2018; Guo et al., 2022). However, facts can only be assessed relative to the knowledge present in the database.”
From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Thorne et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 3×
“, 2018) and WordNet (Miller, 1995) and Ohio State report using vector representations of named entities.”
BackgroundMethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Decomposable Attention Model for Natural Language Inference
Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1244 · Cited by 1,351
Extended Boolean information retrieval
Salton, G., Fox, E. A., & Wu, H. (1983) · Communications of the ACM · doi:10.1145/182.358466 · Cited by 1,058
SciTaiL: A Textual Entailment Dataset from Science Question Answering
Khot, T., Sabharwal, A., & Clark, P. E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12022 · Cited by 419
Recognizing textual entailment: Rational, evaluation and approaches
Dagan, I., Dolan, B., Magnini, B., & Roth, D. (2009) · Natural Language Engineering · doi:10.1017/s1351324909990209 · Cited by 194
Research works printed but not linked: 12
- Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough to beat baseline for sentence em- beddings. In Iclr, pages 1-14.
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open- Domain Questions. Proc. of ACL'17.
- Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics, pages 1657-1668, Vancouver, Canada.
- Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark.
- Matt J Kusner, Yu Sun, Nicholas I Kolkin, and Kilian Q Weinberger. 2015. From Word Embeddings To Doc- ument Distances. Proceedings of The 32nd Inter- national Conference on Machine Learning, 37:957- 966.
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of NAACL-HLT, pages 2227-2237, New Orleans, Louisiana.
- Alec Radford and Tim Salimans. 2018. Improv- ing Language Understanding by Generative Pre- Training. arXiv, pages 1-12.
- Vladimir Svetnik, Andy Liaw, Christopher Tong, J Christopher Culberson, Robert P Sheridan, and Bradley P Feuston. 2003. Random forest: a clas- sification and regression tool for compound classi- fication and qsar modeling. Journal of chemical in- formation and computer sciences, 43(6):1947-1958.
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT.
- Yi Yang and Ming-Wei Chang. 2015. S-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 504-513, Beijing, China.
- Wenpeng Yin, Hinrich Schütze, and Dan Roth. 2018. End-Task Oriented Textual Entailment via Deep Ex- plorations of Inter-Sentence Interactions. In Pro- ceedings of the 56th Annual Meeting of the Associ- ation for Computational Linguistics (Short Papers), pages 540-545, Melbourne, Australia.
Cited by
Sorted by citations
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557
A Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500
Self-citationApproximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.200 · Cited by 357
Self-citation
Metadata
- Authors
- Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A.
- Affiliations
- University of Sheffield (GB)
- Imperial College London (GB)
- Amazon (United States) (US)
- Venue
- Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)
- Published
- 2018
- Pages
- 1-9
- OpenAlex
- W2902421513
- Cited by
- 212
- References
- 6
- Citations by year
- 2026: 7 · 2025: 19 · 2024: 21 · 2023: 27 · 2022: 32 · 2021: 53 · 2020: 27 · 2019: 25 · 2018: 1
- FWCI
- 10.82
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Fact Extraction and VERification Workshop (FEVER) (publishedVersion, cc-by)
- arXiv (Cornell University) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Horizon 2020 Framework Programme (EU‐H2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- fact-checking, fact extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
