CoChat
Sign Up
Cited by 26Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

WiCE: Real-world entailment for claims in Wikipedia

Open access
Kamoi, Ryo, Goyal, Tanya, Rodriguez, Juan, Durrett, Greg·2023·Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2023.emnlp-main.470
Open this paper in CoChat — free accountOpen PDF

Abstract

Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation.However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result.We propose WICE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia.In addition to standard claim-level entailment, WICE provides entailment judgments over subsentence units of the claim, and a minimal subset of evidence sentences that support each subclaim.To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models' performance on multiple datasets at test time.Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address. 1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.emnlp-main.470

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • WICE is a textual entailment dataset that consists of naturally occurring claims from Wikipedia and their cited documents.

    “WICE (Kamoi et al., 2023) is a textual entailment dataset that consists of naturally occurring claims from Wikipedia and their cited documents.”

    From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Kamoi et al., 2023

  • Decomposing a piece of long-form text into shorter sentences or search queries for factuality evaluation is commonly implemented in previous works.

    “Decomposing a piece of long-form text into shorter sentences or search queries for factuality evaluation is commonly implemented in previous works (Kamoi et al., 2023; Gao et al., 2023; Wang et al., 2023; Min et al., 2023; Chern et al., 2023; Wanner et al., 2024; Wei et al., 2024; Guan et al., 2024; Chen et al., 2024).”

    From VeriScore: Evaluating the factuality of verifiable claims in long-form text generation (2024) · cites this paper as Kamoi et al., 2023

  • Decomposition of sentences into atomic facts is not necessary to achieve MiniCheck's high performance.

    “Furthermore, we can do this without a separate claim decomposition step.”

    From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Kamoi et al., 2023

    The “2.1 Claim-Split using LLMs” section describes decomposing claims before annotation, and Appendix D.1 describes a separate decomposition prompt. Our copy does not discuss MiniCheck or whether its performance can be achieved without a separate claim-decomposition step (searched: “without a separate claim decomposition step” and “MiniCheck high performance without decomposition atomic facts”). (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 55 · Cited by 26

Research works printed but not linked: 43

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics. · DOI · Paper page
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  3. Jiangjie Chen, Qiaoben Bao, Changzhi Sun, Xinbo Zhang, Jiaze Chen, Hao Zhou, Yanghua Xiao, and Lei Li. 2022. Loren: Logic-regularized reason- ing for interpretable fact verification. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10482-10491. · DOI
  4. Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can NLI models verify QA systems' predictions? In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 3841-3854, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI
  5. Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, and Tal Schuster. 2023. PropSegmEnt: A large- scale corpus for proposition-level segmentation and entailment recognition. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 8874-8893, Toronto, Canada. Association for Com- putational Linguistics. · DOI
  6. Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sen- tences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461. · DOI · Paper page
  7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proceedings of the First Inter- national Conference on Machine Learning Chal- lenges: Evaluating Predictive Uncertainty Visual Ob- ject Classification, and Recognizing Textual Entail- ment, MLCW'05, page 177-190, Berlin, Heidelberg. Springer-Verlag. · DOI
  8. Sarkar Snigdha Sarathi Das, Chirag Shah, Mengting Wan, Jennifer Neville, Longqi Yang, Reid Ander- sen, Georg Buscher, and Tara Safavi. 2023. S3- dst: Structured open-domain dialogue segmentation and state tracking in the era of llms. arXiv preprint arXiv:2309.08827. · arXiv
  9. Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark. Transactions of the Association for Computational Linguistics, 10:1066-1083. · DOI
  10. Ori Ernst, Ori Shapira, Ramakanth Pasunuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, and Ido Dagan. 2021. Summary-source proposition-level alignment: Task, datasets and supervised baseline. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 310-322, Online. Association for Computational Linguistics. · DOI
  11. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  12. Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1801-1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  13. Tanya Goyal and Greg Durrett. 2020. Evaluating Fac- tuality in Generation with Dependency-level En- tailment. In Conference on Empirical Methods in Natural Language Processing (EMNLP), page 3592-3603. · DOI · Paper page
  14. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation. In Proceedings of the The 2022 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 351-358, Abu Dhabi, UAE. Association for Computational Linguistics.
  15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
  16. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Us- ing natural language to annotate natural language. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Compu- tational Linguistics. · DOI
  17. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3905-3920, Seattle, United States. Association for Computational Lin- guistics. · DOI
  18. Ryo Kamoi, Tanya Goyal, and Greg Durrett. 2023. Shortcomings of Question Answering Based Factual- ity Frameworks for Error Localization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 132-146, Dubrovnik, Croatia. Association for Computational Linguistics.
  19. Yuta Koreeda and Christopher Manning. 2021. Con- tractNLI: A dataset for document-level natural lan- guage inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907-1919, Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI
  20. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In Interna- tional Conference on Learning Representations.
  21. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. Wanli: Worker and ai collaboration for natural language inference dataset creation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv preprint arXiv:1907.11692. · arXiv
  23. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Liny- ong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023. Revisiting the gold standard: Ground- ing summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4140-4170, Toronto, Canada. Association for Computational Linguistics. · DOI
  24. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceed- ings of the Eight International Conference on Com- putational Semantics, pages 140-156, Tilburg, The Netherlands. Association for Computational Linguis- tics.
  25. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  26. Ani Nenkova and Rebecca Passonneau. 2004. Evaluat- ing content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chap- ter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Mas- sachusetts, USA. Association for Computational Lin- guistics.
  27. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining Fact Extraction and Verification with Neural Semantic Matching Networks. In Proceed- ings of the Thirty-Third AAAI Conference on Arti- ficial Intelligence and Thirty-First Innovative Ap- plications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'19/IAAI'19/EAAI'19. AAAI Press. · DOI
  28. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  29. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  30. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learn- ing Library. In Advances in Neural Information Pro- cessing Systems, volume 32.
  31. Fabio Petroni, Samuel Broscheit, Aleksandra Pik- tus, Patrick Lewis, Gautier Izacard, Lucas Hos- seini, Jane Dwivedi-Yu, Maria Lomeli, Timo Schick, Michele Bevilacqua, Pierre-Emmanuel Mazaré, Ar- mand Joulin, Edouard Grave, and Sebastian Riedel. 2023. Improving wikipedia verifiability with ai. Na- ture Machine Intelligence, 5(10):1142-1148. · DOI
  32. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In Proceedings of the Seventh Joint Confer- ence on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  33. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624-643, Online. As- sociation for Computational Linguistics. · DOI · Paper page
  34. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  35. Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ra- makanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 682-687, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI
  36. Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI
  37. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  38. Robert Truswell. 2019. The Oxford handbook of event structure. Oxford University Press.
  39. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics. · DOI
  40. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics. · DOI · Paper page
  41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics. · DOI · Paper page
  42. Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document- level natural language inference. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4913-4922, Online. Association for Computational Linguistics. · DOI
  43. Shiyue Zhang and Mohit Bansal. 2021. Finding a bal- anced degree of automation for summary evaluation. · DOI

Authors
Kamoi, R., Goyal, T., Rodriguez, J., & Durrett, G.
Affiliations
Applied Research Laboratories, The University of Texas at Austin (US)
Corresponding author
Kamoi, Ryo
ORCID
Venue
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2023
Pages
7561-7583
DOI
10.18653/v1/2023.emnlp-main.470
OpenAlex
W4389519041
Cited by
26
References
55
Citations by year
2026: 6 · 2025: 12 · 2024: 4 · 2023: 4
FWCI
4.61
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2145280, CAREER)
  • University of Texas at Austin
  • Open Philanthropy Project
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
natural language inference, claim decomposition, evidence retrieval, fact-checking, GPT-3.5, domain shift

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review