CoChat
Sign Up
Cited by 73Topic ModelingNatural Language Processing TechniquesSentiment Analysis and Opinion Mining

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Open access
Belinkov, Yonatan, Poliak, Adam, Shieber, Stuart, Van Durme, Benjamin, Rush, Alexander·2019·Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
DOI 10.18653/v1/p19-1084
Open this paper in CoChat — free accountOpen PDF

Abstract

Natural Language Inference (NLI) datasets often contain hypothesis-only biases-artifacts that allow models to achieve non-trivial performance without learning whether a premise entails a hypothesis.We propose two probabilistic methods to build models that are more robust to such biases and better transfer across datasets.In contrast to standard approaches to NLI, our methods predict the probability of a premise given a hypothesis and NLI label, discouraging models from ignoring the premise.We evaluate our methods on synthetic and existing NLI datasets by training on datasets containing biases and testing on datasets containing no (or different) hypothesis-only biases.Our results indicate that these methods can make NLI models more robust to dataset-specific artifacts, transferring better than a baseline architecture in 9 out of 12 NLI datasets.Additionally, we provide an extensive analysis of the interplay of our methods with known biases in NLI datasets, as well as the effects of encouraging models to ignore biases and fine-tuning on target datasets.1 * * Equal contribution 1 Our code is available at https://github.com/ azpoliak/robust-nli.2 This hypothesis contradicts the premise and would likely not be inferred.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p19-1084

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Deep Neural Networks have been proven to easily learn biases from the data, which significantly impacts their reliability.

    “Deep Neural Networks have been proven to easily learn biases from the data, which significantly impacts their reliability.”

    From Large Language Models are not Fair Evaluators (2024) · cites this paper as Belinkov et al., 2019

    The Synthetic Experiments section (5.1; page not provided) reports that a baseline NLI model learned a bias in its training data and failed to generalize to an unbiased test set. The Introduction (section 1; page not provided) also links ignoring biases to better transfer. These findings support the concern about reliability, but the claim extends an NLI-specific result to deep neural networks generally. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 64

  1. Martín Abadi and David G. Andersen. Learning to protect communications with adversarial neu- ral cryptography. arXiv, 2016. URL https:// arxiv.org/abs/1610.06918. · arXiv
  2. Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Ques- tion Answering. arXiv preprint arXiv:1712.00377, 2017. · arXiv
  3. Yonatan Belinkov and Yonatan Bisk. Synthetic and Natural Noise Both Break Neural Machine Trans- lation. In International Conference on Learn- ing Representations, 2018. URL https:// openreview.net/forum?id=BJ8vJebC-.
  4. Yonatan Belinkov and James Glass. Analysis Methods in Neural Language Processing: A Survey. Trans- actions of the Association for Computational Lin- guistics (TACL), 7:49-72, 2019. doi: 10.1162/tacl\ a\ 00254. URL https://doi.org/10.1162/ tacl a 00254. · DOI
  5. Yonatan Belinkov, Adam Poliak, Stuart M. Shieber, Benjamin Van Durme, and Alexander Rush. On Ad- versarial Removal of Hypothesis-only Bias in Nat- ural Language Inference. In Proceedings of the Eighth Joint Conference on Lexical and Computa- tional Semantics (*SEM, Oral presentation), June 2019.
  6. Yoshua Bengio and Jean-Sébastien Senecal. Quick Training of Probabilistic Neural Nets by Importance Sampling. In Proceedings of the Ninth Interna- tional Workshop on Artificial Intelligence and Statistics, AISTATS 2003, Key West, Florida, USA, January 3-6, 2003, pp. 1-9, 2003. URL http: //research.microsoft.com/en-us/ um/cambridge/events/aistats2003/ proceedings/164.pdf.
  7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated cor- pus for learning natural language inference. In Pro- ceedings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). As- sociation for Computational Linguistics, 2015.
  8. Zheng Cai, Lifu Tu, and Kevin Gimpel. Pay At- tention to the Ending: Strong Neural Baselines for the ROC Story Cloze Task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pp. 616-622. Association for Computational Linguistics, 2017. doi: 10.18653/v1/P17-2097. URL http://www.aclweb.org/anthology/ P17-2097. · DOI
  9. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1657-1668. Association for Compu- tational Linguistics, 2017. doi: 10.18653/v1/ P17-1152. URL http://www.aclweb.org/ anthology/P17-1152. · DOI
  10. Jianpeng Cheng, Li Dong, and Mirella Lapata. Long Short-Term Memory-Networks for Machine Reading. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Lan- guage Processing, pp. 551-561, Austin, Texas, November 2016. Association for Computational Linguistics. URL https://aclweb.org/ anthology/D16-1053.
  11. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Supervised Learning of Universal Sentence Representations from Natu- ral Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pp. 670-680. Association for Computational Linguistics, 2017. URL http: //aclweb.org/anthology/D17-1070.
  12. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment chal- lenge. In Machine learning challenges. evaluat- ing predictive uncertainty, visual object classifica- tion, and recognising tectual entailment, pp. 177- 190. Springer, 2006.
  13. Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, and Tiejun Zhao. Attention-Fused Deep Matching Network for Natural Language In- ference. In Proceedings of the Twenty-Seventh Inter- national Joint Conference on Artificial Intelligence, IJCAI-18, pp. 4033-4040. International Joint Con- ferences on Artificial Intelligence Organization, 7 2018. doi: 10.24963/ijcai.2018/561. URL https: //doi.org/10.24963/ijcai.2018/561. · DOI
  14. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. On Adversarial Examples for Character-Level Neural Machine Translation. In Proceedings of the 27th In- ternational Conference on Computational Linguis- tics, pp. 653-663. Association for Computational Linguistics, 2018. URL http://aclweb.org/ anthology/C18-1055.
  15. Yanai Elazar and Yoav Goldberg. Adversarial Re- moval of Demographic Attributes from Text Data. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 11-21. Association for Computational Lin- guistics, 2018. URL http://aclweb.org/ anthology/D18-1002.
  16. Yaroslav Ganin and Victor Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In In- ternational Conference on Machine Learning, pp. 1180-1189, 2015.
  17. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Lavi- olette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1): 2096-2030, 2016.
  18. Max Glockner, Vered Shwartz, and Yoav Goldberg. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pp. 650-655, Melbourne, Australia, July 2018. Associ- ation for Computational Linguistics. URL http: //www.aclweb.org/anthology/P18-2103.
  19. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image under- standing in Visual Question Answering. In CVPR, volume 1, pp. 3, 2017.
  20. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Asso- ciation for Computational Linguistics. URL http: //www.aclweb.org/anthology/N18-2017.
  21. Michael Gutmann and Aapo Hyvärinen. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artifi- cial Intelligence and Statistics, pp. 297-304, 2010.
  22. Georg Heigold, Günter Neumann, and Josef van Gen- abith. How Robust Are Character-Based Word Em- beddings in Tagging and MT Against Wrod Scraml- bing or Randdm Nouse? In Proceedings of the 13th Conference of The Association for Machine Transla- tion in the Americas (Volume 1: Research Track, pp. 68-79, March 2018.
  23. Robin Jia and Percy Liang. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pp. 2011-2021, Copenhagen, Denmark, Septem- ber 2017. URL https://www.aclweb.org/ anthology/D17-1215.
  24. K. Kafle and C. Kanan. Answer-Type Prediction for Visual Question Answering. In 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 4976-4984, June 2016. doi: 10.1109/ CVPR.2016.538. · DOI
  25. Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Exam- ples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 2418-2428, Melbourne, Australia, July 2018. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P18-1225.
  26. Divyansh Kaushik and Zachary C. Lipton. How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pp. 5010-5015, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ D18-1546.
  27. Tushar Khot, Ashish Sabharwal, and Peter Clark. Sc- iTail: A Textual Entailment Dataset from Science Question Answering. In AAAI, 2018.
  28. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. Nat- ural Language Inference from Multiple Premises. In Proceedings of the Eighth International Joint Con- ference on Natural Language Processing (Volume 1: Long Papers), pp. 100-109, Taipei, Taiwan, Novem- ber 2017. Asian Federation of Natural Language Processing. URL http://www.aclweb.org/ anthology/I17-1011.
  29. Yitong Li, Timothy Baldwin, and Trevor Cohn. To- wards Robust and Privacy-preserving Text Repre- sentations. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 25-30. Association for Computational Linguistics, 2018. URL http: //aclweb.org/anthology/P18-2005.
  30. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. Learning natural language inference using bidirec- tional LSTM model and inner-attention. arXiv preprint arXiv:1605.09090, 2016. · arXiv
  31. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella bernardi, and Roberto Zam- parelli. A SICK cure for the evaluation of composi- tional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pp. 216-223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). ISBN 978-2-9517408-8-4. URL http: //www.lrec-conf.org/proceedings/ lrec2014/pdf/363 Paper.pdf. ACL Anthology Identifier: L14-1314.
  32. Pasquale Minervini and Sebastian Riedel. Adversar- ially Regularising Neural NLI Models to Integrate Logical Background Knowledge. In Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018). Association for Computational Linguistics, 2018.
  33. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 839- 849, San Diego, California, June 2016. Associa- tion for Computational Linguistics. URL http: //www.aclweb.org/anthology/N16-1098.
  34. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural Language Inference by Tree-Based Convolution and Heuristic Matching. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pp. 130-136, Berlin, Germany, Au- gust 2016. Association for Computational Linguis- tics. URL http://anthology.aclweb.org/ P16-2022.
  35. Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. Did the Model Understand the Question? In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1896-1906. Association for Computational Lin- guistics, 2018. URL http://aclweb.org/ anthology/P18-1176.
  36. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A Decomposable Attention Model for Natural Language Inference. In Proceed- ings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2249- 2255. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1244. URL http: //www.aclweb.org/anthology/D16-1244. · DOI
  37. Ellie Pavlick and Chris Callison-Burch. Most "ba- bies" are "little" and most "problems" are "huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2164-2173. Association for Computational Linguistics, 2016. doi: 10.18653/ v1/P16-1204. URL http://www.aclweb.org/ anthology/P16-1204. · DOI
  38. Ellie Pavlick, Travis Wolfe, Pushpendre Rastogi, Chris Callison-Burch, Mark Dredze, and Benjamin Van Durme. FrameNet+: Fast Paraphrastic Tripling of FrameNet. In Proceedings of the 53rd An- nual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pp. 408-413, Beijing, China, July 2015. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ P15-2067.
  39. Jeffrey Pennington, Richard Socher, and Christo- pher D. Manning. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532- 1543, 2014. URL http://www.aclweb.org/ anthology/D14-1162.
  40. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting Diverse Natu- ral Language Inference Problems for Sentence Rep- resentation Evaluation. In Empirical Methods in Natural Language Processing (EMNLP), 2018a.
  41. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hy- pothesis Only Baselines in Natural Language In- ference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Seman- tics, pp. 180-191, New Orleans, Louisiana, June 2018b. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ S18-2023.
  42. Altaf Rahman and Vincent Ng. Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge. In Proceedings of the 2012 Joint Con- ference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 777-789, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ D12-1071.
  43. Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Sys- tems, pp. 1548-1558, 2018.
  44. Pushpendre Rastogi and Benjamin Van Durme. Sublinear partition estimation. arXiv preprint arXiv:1508.01596, 2015. · arXiv
  45. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. Semantic Proto-Roles. Transactions of the Association for Computational Linguistics, 3:475- 488, 2015. ISSN 2307-387X. URL https: //tacl2013.cs.columbia.edu/ojs/ index.php/tacl/article/view/674.
  46. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically Equivalent Adversarial Rules for Debugging NLP models. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 856-865. Association for Computational Linguistics, 2018. URL http://aclweb.org/ anthology/P18-1079.
  47. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. Rea- soning about Entailment with Neural Attention. In International Conference on Learning Representa- tions (ICLR), 2016.
  48. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task. In Pro- ceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 15-25. Association for Computational Linguistics, 2017a. doi: 10.18653/v1/K17-1004. URL http: //www.aclweb.org/anthology/K17-1004. · DOI
  49. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. Story Cloze Task: UW NLP System. In Proceedings of LSDSem, 2017b.
  50. Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the Story Ending Biases in The Story Cloze Test. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 2: Short Papers), pp. 752-757, Melbourne, Australia, July 2018. Associ- ation for Computational Linguistics. URL http: //www.aclweb.org/anthology/P18-2119.
  51. Rion Snow, Lucy Vanderwende, and Arul Menezes. Effectively Using Syntax for Recognizing False En- tailment. In Proceedings of the Human Language Technology Conference of the NAACL, Main Con- ference, 2006. URL http://www.aclweb.org/ anthology/N06-1005.
  52. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural net- works. In International Conference on Learning Representations (ICLR), 2014.
  53. Aarne Talman and Stergios Chatzikyriakidis. Neu- ral Network Models for Natural Language Infer- ence Fail to Capture the Semantics of Inference. CoRR, abs/1810.09774, 2018. URL http:// arxiv.org/abs/1810.09774. · arXiv
  54. Florian Tramr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble Adversarial Training: Attacks and De- fenses. In International Conference on Learn- ing Representations, 2018. URL https:// openreview.net/forum?id=rkZvSe-RZ.
  55. Lucy Vanderwende and William B Dolan. What syn- tax can contribute in the entailment task. In Machine Learning Challenges. Evaluating Predictive Uncer- tainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 205-216. Springer, 2006.
  56. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ W18-5446.
  57. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. Inference is Everything: Recasting Semantic Resources into a Unified Evalu- ation Framework. In Proceedings of the Eighth In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 996-1005, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL http:// www.aclweb.org/anthology/I17-1100.
  58. Adina Williams, Nikita Nangia, and Samuel Bow- man. A Broad-Coverage Challenge Corpus for Sen- tence Understanding through Inference. In Proceed- ings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL http: //aclweb.org/anthology/N18-1101.
  59. Qile Zhu Xiaolin Li Xiaoyong Yuan, Pan He. Ad- versarial Examples: Attacks and Defenses for Deep Learning. arXiv preprint arXiv:1712.07107, 2017. · arXiv
  60. Chen Zhang. Natural Language Interference from Tex- tual Entailment to Conversation Entailment. PhD thesis, Michigan State University, East Lansing, MI, USA, 2010. AAI3435149.
  61. Chen Zhang and Joyce Chai. Towards Conversa- tion Entailment: An Empirical Investigation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 756- 766, Cambridge, MA, October 2010. Association for Computational Linguistics. URL https:// www.aclweb.org/anthology/D10-1074.
  62. Chen Zhang and Joyce Y. Chai. What Do We Know About Conversation Participants: Experi- ments on Conversation Entailment. In Proceed- ings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL '09, pp. 206-215, Stroudsburg, PA, USA, 2009. Associa- tion for Computational Linguistics. ISBN 978- 1-932432-64-0. URL http://dl.acm.org/ citation.cfm?id=1708376.1708406.
  63. P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and Yang: Balancing and Answer- ing Binary Visual Questions. In 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 5014-5022, June 2016. doi: 10.1109/ CVPR.2016.542. · DOI
  64. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Ben- jamin Van Durme. Ordinal Common-sense Infer- ence. Transactions of the Association for Compu- tational Linguistics, 5:379-395, 2017. ISSN 2307- 387X. URL https://transacl.org/ojs/ index.php/tacl/article/view/1082.

Cited by

Sorted by citations

Authors
Belinkov, Y., Poliak, A., Shieber, S., Van Durme, B., & Rush, A.
Affiliations
Harvard University (US)
ORCID
Venue
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Publisher
Association for Computational Linguistics
Published
2019
Pages
877-891
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/p19-1084
OpenAlex
W2952341934
Cited by
73
References
2
Citations by year
2026: 2 · 2025: 1 · 2024: 6 · 2023: 15 · 2022: 17 · 2021: 16 · 2020: 14 · 2019: 2
FWCI
4.69
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1845664)
  • Defense Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
natural language inference, stochastic modeling, cross-dataset transfer, robustness, logical consequence, fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review