CoChat
Sign Up
Cited by 32Topic ModelingNatural Language Processing TechniquesMachine Learning and Data Classification

The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

Open access
Zhou, Xiang, Nie, Yixin, Tan, Hao, Bansal, Mohit·2020·Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
DOI 10.18653/v1/2020.emnlp-main.659
Open this paper in CoChat — free accountOpen PDF

Abstract

We find that the performance of state-of-theart models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable.This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from?(3) How should we handle this instability and what are some potential solutions?For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve.We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine.Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets.Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models. 1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.emnlp-main.659

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work has shown that there is often insufficient information at training time to distinguish models that would generalize well under distribution shift; many models that perform similarly in-distribution (ID) can vary substantially out-of-distribution (OOD).

    “Prior work has shown that there is often insufficient information at training time to distinguish models that would generalize well under distribution shift; many models that perform similarly in-distribution (ID) can vary substantially out-of-distribution (OOD) (McCoy et al., 2019a; Zhou et al., 2020; D'Amour et al., 2020a).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Zhou et al., 2020

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Michael Bloodgood and John Grothendieck. 2013. Analysis of stopping active learning based on sta- bilizing predictions. In Proceedings of the Seven- teenth Conference on Computational Natural Lan- guage Learning, pages 10-19.
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642.
  3. Vicente Iván Sánchez Carmona, Jeff Mitchell, and Se- bastian Riedel. 2018. Behavior analysis of nli mod- els: Uncovering the influence of three factors on ro- bustness. NAACL.
  4. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1657-1668.
  5. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Car- bonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2978-2988.
  6. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stop- ping. arXiv preprint arXiv:2002.06305. · arXiv
  7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language process- ing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6.
  8. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Chris Potts. 2019. Posing fair generalization tasks for natural language inference. EMNLP.
  9. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that re- quire simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 2: Short Papers), pages 650-655.
  10. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image under- standing in visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913.
  11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112.
  12. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142.
  13. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question an- swering baselines. In European conference on com- puter vision, pages 727-739. Springer.
  14. Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. EMNLP.
  15. Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension re- quire? a critical investigation of popular bench- marks. EMNLP.
  16. Mike Lewis and Angela Fan. 2019. Generative ques- tion answering: Learning to answer the whole ques- tion. In ICLR.
  17. Nelson F Liu, Roy Schwartz, and Noah A Smith. 2019a. Inoculation by fine-tuning: A method for analyzing challenge datasets. NAACL.
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  19. Matthew C Makel, Jonathan A Plucker, and Boyd Hegarty. 2012. Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6):537-542.
  20. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compo- sitional distributional semantic models. In LREC.
  21. R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019a. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969. · arXiv
  22. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019b. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448.
  23. Nafise Sadat Moosavi, Prasetya Ajie Utama, Andreas Rücklé, and Iryna Gurevych. 2019. Improving gen- eralization by incorporating coverage in natural lan- guage inference. arXiv preprint arXiv:1909.08940. · arXiv
  24. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018a. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353.
  25. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018b. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Com- putational Linguistics.
  26. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. Analyzing compositionality-sensitivity of nli mod- els. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 33, pages 6867-6874.
  27. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Ad- versarial nli: A new benchmark for natural language understanding. ACL.
  28. Deric Pang, Lucy H Lin, and Noah A Smith. 2019. Im- proving natural language inference with a pretrained parser. arXiv preprint arXiv:1909.08217. · arXiv
  29. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In NAACL.
  30. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. · arXiv
  31. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for squad. ACL.
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. SQuAD: 100,000+ questions for machine comprehension of text. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Lin- guistics. · DOI · Paper page
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. Squad: 100,000+ questions for machine comprehension of text. EMNLP.
  34. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. Equate: A bench- mark evaluation framework for quantitative reason- ing in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Lan- guage Learning (CoNLL), pages 349-361.
  35. Nils Reimers and Iryna Gurevych. 2017. Report- ing score distributions makes a difference: Perfor- mance study of lstm-networks for sequence tagging. EMNLP.
  36. Nils Reimers and Iryna Gurevych. 2018. Why com- paring single performance scores does not allow to draw conclusions about machine learning ap- proaches. arXiv preprint arXiv:1803.09578. · arXiv
  37. Vered Shwartz and Ido Dagan. 2018. Paraphrase to explicate: Revealing implicit noun-compound rela- tions. ACL.
  38. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recogniz- ing textual entailment. LREC.
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
  40. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of ICLR.
  41. Noah Weber, Leena Shekhar, and Niranjan Balasubra- manian. 2018. The fine line between linguistic gen- eralization and failure in seq2seq-attention models. NAACL.
  42. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
  43. Yadollah Yaghoobzadeh, Remi Tachet, TJ Hazen, and Alessandro Sordoni. 2019. Robust natural language inference models with example forgetting. arXiv preprint arXiv:1911.03861. · arXiv
  44. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. NeurIPS.
  45. Yi-Ting Yeh and Yun-Nung Chen. 2019. Qainfomax: Learning robust question answering system by mu- tual information maximization. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3361-3366.

Cited by

Sorted by citations

Authors
Zhou, X., Nie, Y., Tan, H., & Bansal, M.
Affiliations
University of North Carolina at Chapel Hill (US)
ORCID
Venue
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Publisher
Association for Computational Linguistics
Published
2020
Pages
8215-8228
DOI
10.18653/v1/2020.emnlp-main.659
OpenAlex
W3104492344
Cited by
32
References
4
Citations by year
2024: 3 · 2023: 4 · 2022: 9 · 2021: 6 · 2020: 10
FWCI
2.87
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1846185, CAREER)
  • Defense Advanced Research Projects Agency
  • Office of Naval Research (N000141812871, N00014-18, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
performance instability, natural language inference, reading comprehension, model selection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review