CoChat
Sign Up
Cited by 51Topic ModelingMisinformation and Its ImpactsExplainable Artificial Intelligence (XAI)

ProoFVer: Natural Logic Theorem Proving for Fact Verification

Open access
Krishna, Amrith, Riedel, Sebastian, Vlachos, Andreas·2022·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00503
Open this paper in CoChat — free account

Abstract

Fact verification systems typically rely on neural network classifiers for veracity prediction, which lack explainability. This paper proposes ProoFVer, which uses a seq2seq model to generate natural logic-based inferences as proofs. These proofs consist of lexical mutations between spans in the claim and the evidence retrieved, each marked with a natural logic operator. Claim veracity is determined solely based on the sequence of these operators. Hence, these proofs are faithful explanations, and this makes ProoFVer faithful by construction. Currently, ProoFVer has the highest label accuracy and the second best score in the FEVER leaderboard. Furthermore, it improves by 13.21% points over the next best model on a dataset with counterfactual instances, demonstrating its robustness. As explanations, the proofs show better overlap with human rationales than attention-based highlights and the proofs help humans predict model decisions correctly more often than using the evidence directly.1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00503

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. Lasha Abzianidze. 2017a. LangPro: Natural language theorem prover. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 115-120, Copenhagen, Denmark. Association for Computational Linguistics.
  2. Lasha Abzianidze. 2017b. A Natural Proof System for Natural Language . Ph.D. thesis, Tilburg University.
  3. Naser Ahmadi, Joohyung Lee, Paolo Papotti, and Mohammed Saeed. 2019. Explainable fact checking with probabilistic answer set programming. In Proceedings of the 2019 Truth and Trust Online Conference (TTO 2019), London, UK, October 4-5, 2019 .
  4. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) , pages 5459, Minneapolis, Minnesota. Association for Computational Linguistics.
  5. Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , volume 1.
  6. Gabor Angeli. 2016. Learning Open Domain Knowledge From Text . Ph.D. thesis, Stanford University.
  7. Gabor Angeli and Christopher D. Manning. 2014. NaturalLI: Natural logic inference for common sense reasoning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 534545, Doha, Qatar. Association for Computational Linguistics.
  8. Gabor Angeli, Neha Nayak, and Christopher D. Manning. 2016. Combining natural logic and shallow reasoning for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 442-452, Berlin, Germany. Association for Computational Linguistics.
  9. Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7352-7364, Online. Association for Computational Linguistics.
  10. Roy Bar-Haim, Ido Dagan, Iddo Greental, Idan Szpektor, and Moshe Friedman. 2007. Semantic inference at the lexical-syntactic level for textual entailment recognition. In Proceedings of the ACL-PASCAL Workshop on Tex- tual Entailment and Paraphrasing , pages 131136, Prague. Association for Computational Linguistics.
  11. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representations .
  12. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4443-4458, Online. Association for Computational Linguistics.
  13. Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. 2020. Generating fact checking briefs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7147-7161, Online. Association for Computational Linguistics.
  14. Yufei Feng, Zi'ou Zheng, Quan Liu, Michael Greenspan, and Xiaodan Zhu. 2020. Exploring end-to-end differentiable natural logic modeling. In Proceedings of the 28th International Conference on Computational Linguistics , pages 1172-1185, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  15. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin , 76(5):378-382.
  16. Robert M. French. 1999. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences , 3(4):128-135.
  17. Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, and Gerhard Weikum. 2019. Exfakt: A framework for explaining facts over knowledge graphs and text. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining , WSDM '19, page 87-95,
  18. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 758764, Atlanta, Georgia. Association for Computational Linguistics.
  19. Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020. Explaining black box predictions and unveiling data artifacts through influence functions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5553-5563, Online. Association for Computational Linguistics.
  20. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) , pages 493-503, Hong Kong, China. Association for Computational Linguistics.
  21. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-athene: Multisentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) , pages 103-108, Brussels, Belgium. Association for Computational Linguistics.
  22. Stefan Harmeling. 2009. Inferring textual entailment with a probabilistically sound calculus. Natural Language Engineering , 15(4):459-477.
  23. Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 1011-1019, Los Angeles, California. Association for Computational Linguistics.
  24. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '04, page 168-177, New York, NY, USA. Association for Computing Machinery.
  25. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4198-4205, Online. Association for Computational Linguistics.
  26. Alon Jacovi and Yoav Goldberg. 2021. Aligning Faithful Interpretations with their Social Attribution. Transactions of the Association for Computational Linguistics , 9:294-310.
  27. Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to faithfully rationalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4459-4473, Online. Association for Computational Linguistics.
  28. Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings , pages 1627-1643, Online. Association for Computational Linguistics.
  29. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing , pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
  30. Isaac Johnson. 2020. Analyzing wikidata transclusion on english wikipedia. In Proceedings of the 1st Wikidata Workshop (Wikidata 2020) co-located with 19th International Semantic Web Conference(OPub 2020), Virtual Conference, November 2-6, 2020 , volume 2773 of CEUR Workshop Proceedings . CEUR-WS.org.
  31. Lauri Karttunen. 2015. From natural logic to natural reasoning. In Computational Linguistics and Intelligent Text Processing , pages 295-309, Cham. Springer International Publishing.
  32. Neema Kotonya and Francesca Toni. 2020a. Explainable automated fact-checking: A survey. In Proceedings of the 28th International Conference on Computational Linguistics , pages 5430-5443, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  33. Neema Kotonya and Francesca Toni. 2020b. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7740-7754, Online. Association for Computational Linguistics.
  34. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 107-117, Austin, Texas. Association for Computational Linguistics.
  35. Percy Liang, Michael I. Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics , 39(2):389-446.
  36. Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7342-7351, Online. Association for Computational Linguistics.
  37. Bill MacCartney. 2009. Natural Language Inference . Ph.D. thesis, Stanford University.
  38. Bill MacCartney and Christopher D. Manning. 2007. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing , pages 193-200, Prague. Association for Computational Linguistics.
  39. Yashar Mehdad. 2009. Automatic cost estimation for tree edit distance using particle swarm optimization. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers , pages 289-292, Suntec, Singapore. Association for Computational Linguistics.
  40. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM , 38(11):39-41.
  41. Ellie Pavlick and Chris Callison-Burch. 2016. Socalled non-subsective adjectives. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics , pages 114119, Berlin, Germany. Association for Computational Linguistics.
  42. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classifica- tion. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 425-430, Beijing, China. Association for Computational Linguistics.
  43. Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018. DeClarE: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 22-32, Brussels, Belgium. Association for Computational Linguistics.
  44. Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 13141324, New Orleans, Louisiana. Association for Computational Linguistics.
  45. Swarnadeep Saha, Prateek Yadav, and Mohit Bansal. 2021. multiPRover: Generating multiple proofs for improved interpretability in rule reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3662-3677, Online. Association for Computational Linguistics.
  46. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 624-643, Online. Association for Computational Linguistics.
  47. Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3419-3425, Hong Kong, China. Association for Computational Linguistics.
  48. Dominik Stammbach. 2021. Evidence selection as a token-level prediction task. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER) , pages 14-20, Dominican Republic. Association for Computational Linguistics.
  49. Asher Stern, Roni Stern, Ido Dagan, and Ariel Felner. 2012. Efficient search for transformationbased inference. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 283-291, Jeju Island, Korea. Association for Computational Linguistics.
  50. Idan Szpektor, Hristo Tanev, Ido Dagan, and Bonaventura Coppola. 2004. Scaling webbased acquisition of entailment relations. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , pages 41-48, Barcelona, Spain. Association for Computational Linguistics.
  51. James Thorne and Andreas Vlachos. 2021a. Elastic weight consolidation for better bias inoculation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 957-964, Online. Association for Computational Linguistics.
  52. James Thorne and Andreas Vlachos. 2021b. Evidence-based factual error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3298-3309, Online. Association for Computational Linguistics.
  53. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
  54. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER) , pages 1-6, Hong Kong, China. Association for Computational Linguistics.
  55. Joseph E. Uscinski and Ryden W. Butler. 2013. The epistemology of fact checking. Critical Review , 25(2):162-180.
  56. V.M.S. Valencia. 1991. Studies on Natural Logic and Categorial Grammar . Ph.D. thesis, University of Amsterdam.
  57. Johan Van Benthem. 1986. Natural logic. In Essays in Logical Semantics , pages 109-119, Dordrecht. Springer Netherlands.
  58. Denny Vrandeˇ ci´ c and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM , 57(10):78-85.
  59. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7534-7550, Online. Association for Computational Linguistics.
  60. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics , 9(0):176-194.
  61. Lianwei Wu, Yuan Rao, Yongqiang Zhao, Hao Liang, and Ambreen Nazir. 2020. DTCA: Decision tree-based co-attention networks for explainable claim verification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1024-1035, Online. Association for Computational Linguistics.
  62. Weijia Xu and Marine Carpuat. 2021. EDITOR: An edit-based transformer with repositioning for neural machine translation with soft lexical constraints. Transactions of the Association for Computational Linguistics , 9:311-328.
  63. Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 71707186, Online. Association for Computational Linguistics.
  64. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems , volume 33, pages 17283-17297. Curran Associates, Inc.
  65. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the TwentyFirst Conference on Uncertainty in Artificial Intelligence , UAI'05, page 658-666, Arlington, Virginia, USA. AUAI Press.

Cited by

Sorted by citations

Authors
Krishna, A., Riedel, S., & Vlachos, A.
Affiliations
  • University of Cambridge (GB)
  • University College London (GB)
Corresponding author
  • Krishna, Amrith
  • Riedel, Sebastian
  • Vlachos, Andreas
ORCID
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2022
Volume
10
Pages
1013-1030
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl_a_00503
OpenAlex
W3194769714
Cited by
51
References
7
Citations by year
2026: 7 · 2025: 11 · 2024: 13 · 2023: 16 · 2022: 4
FWCI
3.62
Type
journal article
Language
en
Open access
Licence
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • UCL Discovery (University College London) (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
Horizon 2020 Framework Programme
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
fact-checking, natural logic, theorem proving, explainable AI, proof generation, sequence-to-sequence models, fever

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review