UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF)
Open accessAbstract
In this paper we describe our 2 nd place FEVER shared-task system that achieved a FEVER score of 62.52% on the provisional test set (without additional human evaluation), and 65.41% on the development set.Our system is a four stage model consisting of document retrieval, sentence retrieval, natural language inference and aggregation.Retrieval is performed leveraging task-specific features, and then a natural language inference model takes each of the retrieved sentences paired with the claimed fact.The resulting predictions are aggregated across retrieved sentences with a Multi-Layer Perceptron, and re-ranked corresponding to the final prediction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most literature leverages Wikipedia as the knowledge source, which is only a small part of world knowledge.
“Most of the literature leverages Wikipedia as the knowledge source [135, 249, 300], which is only a small part of world knowledge.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [300]
The Introduction describes FEVER as using a corpus of Wikipedia articles, and §2 describes this paper’s system retrieving evidence from them. The paper is therefore an example of work using Wikipedia as a knowledge source, but it does not establish that most literature does so or discuss Wikipedia’s share of world knowledge. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Fact Checking: Task definition and dataset construction
Vlachos, A., & Riedel, S. (2014) · Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science · doi:10.3115/v1/w14-2508 · Cited by 402 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 10
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642. Association for Computational Linguis- tics.
- Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1657-1668. Association for Computational Linguis- tics.
- Afonso Mendes, David Nogueira, Samuel Broscheit, Filipe Aleixo, Pedro Balage, Rui Martins, Sebastiao Miranda, and Mariana S. C. Almeida. 2017. Summa at tac knowledge base population task 2017. In Pro- ceedings of the Text Analysis Conference (TAC) KBP 2017, Gaithersburg, Maryland USA. National Insti- tute of Standards and Technology.
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pages 807-814, USA.
- Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable atten- tion model for natural language inference. CoRR, abs/1606.01933. · arXiv
- Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532- 1543.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. CoRR, abs/1802.05365. · arXiv
- Georgios P. Spithourakis and Sebastian Riedel. 2018. Numeracy for language models: Evaluating and im- proving their ability to predict numbers. CoRR, abs/1805.08154. · arXiv
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT.
- David Uberti. 2016. Washington post fake news story blurs the definition of fake news. Columbia Jour- nalism Review.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Reasoning Over Semantic-Level Graph for Fact Checking
Zhong, W., Xu, J., Tang, D., Xu, Z., Duan, N., Zhou, M., Wang, J., & Yin, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.549 · Cited by 151 · see citing works
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
Nie, Y., Wang, S., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1258 · Cited by 132 · see citing works
GERE: Generative Evidence Retrieval for Fact Verification
Chen, J., Zhang, R., Guo, J., Fan, Y., & Cheng, X. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531827 · Cited by 66 · see citing works
- Authors
- Yoneda, T., Mitchell, J., Welbl, J., Stenetorp, P., & Riedel, S.
- Affiliations
- Toyota Technological Institute (JP)
- University College London (GB)
- Venue
- Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)
- Published
- 2018
- Pages
- 97-102
- OpenAlex
- W2955753753
- Cited by
- 101
- References
- 2
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 6 · 2023: 4 · 2022: 16 · 2021: 33 · 2020: 18 · 2019: 18 · 2018: 1
- FWCI
- 7.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Fact Extraction and VERification Workshop (FEVER) (publishedVersion, cc-by)
- Funders
- European Commission (H2020, 688139)
- Horizon 2020 Framework Programme
- Engineering and Physical Sciences Research Council (H2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- fact-checking, document retrieval, sentence retrieval, natural language inference, evidence aggregation, multilayer perceptron, re-ranking, task-specific features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
