CoChat
Sign Up
Cited by 63Natural Language Processing TechniquesTopic ModelingSentiment Analysis and Opinion Mining

Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

Open access
Dror, Rotem, Baumer, Gili, Bogomolov, Marina, Reichart, Roi·2017·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00074
Open this paper in CoChat — free accountOpen PDF

Abstract

With the ever growing amount of textual data from a large variety of languages, domains, and genres, it has become standard to evaluate NLP algorithms on multiple datasets in order to ensure a consistent performance across heterogeneous setups. However, such multiple comparisons pose significant challenges to traditional statistical analysis methods in NLP and can lead to erroneous conclusions. In this paper we propose a Replicability Analysis framework for a statistically sound analysis of multiple comparisons between algorithms for NLP tasks. We discuss the theoretical advantages of this framework over the current, statistically unjustified, practice in the NLP literature, and demonstrate its empirical value across four applications: multi-domain dependency parsing, multilingual POS tagging, cross-domain sentiment classification and word similarity prediction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00074

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A statistically sound way to test for significance under cross-validation is to calculate the p-value for each fold separately and then perform replicability analysis for dependent datasets with K Bonferroni.

    “One statistically sound way to test for significance when a cross-validation protocol is employed is hence to calculate the pvalue for each fold separately, and then to perform replicability analysis for dependent datasets with K Bonf erroni .”

    From The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018) · cites this paper as Dror et al., 2017

    Section 4.1 recommends Bonferroni-based partial conjunction testing for dependent datasets, and the Framework Summary recommends reporting kBonferroni when independence cannot be assumed. Our copy does not establish that cross-validation folds should each receive a separate p-value or that folds should be treated as the datasets in this framework (searched: cross-validation, folds). (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Dror et al. (2017) proposes a statistical significance framework for multiple comparisons performed with dependent test sets, using the K Bonferroni estimator for the number of datasets with significant effect.

    “This paper proposes a statistical significance framework for multiple comparisons performed with dependent test sets, using the K Bonf erroni estimator for the number of datasets with significant effect.”

    From The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018) · cites this paper as Dror et al., 2017

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 73 · Cited by 63

Research works printed but not linked: 68

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pas ¸ca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of HLT-NAACL.
  2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multi- lingual NLP. In Proceedings of CoNLL.
  3. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcatego- rization acquisition. In Proceedings of EMNLP.
  4. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don't count, predict! a systematic compari- son of context-counting vs. context-predicting seman- tic vectors. In Proceedings of ACL.
  5. Yoav Benjamini and Ruth Heller. 2008. Screen- ing for partial conjunction hypotheses. Biometrics, 64(4):1215-1222.
  6. Yoav Benjamini and Yosef Hochberg. 1995. Control- ling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Sta- tistical Society. Series B (Methodological), pages 289- 300.
  7. Yoav Benjamini, Abba M. Krieger, and Daniel Yekutieli. 2006. Adaptive linear step-up procedures that control the false discovery rate. Biometrika, pages 491-507.
  8. Yoav Benjamini, Ruth Heller, and Daniel Yekutieli. 2009. Selective inference in complex research. Philo- sophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 367(1906):4255-4271.
  9. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical signifi- cance in NLP. In Proceedings of EMNLP-CoNLL.
  10. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL.
  11. Bernd Bohnet. 2010. Very high accuracy and fast depen- dency parsing is not a contradiction. In Proceedings of COLING.
  12. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam- Khanh Tran. 2012. Distributional semantics in tech- nicolor. In Proceedings of ACL.
  13. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Arti- ficial Intelligence Research (JAIR), 49:1-47.
  14. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-x shared task on multilingual dependency parsing. In Proceedings of CoNLL.
  15. Yee Seng Chan and Hwee Tou Ng. 2007. Domain adap- tation with active learning for word sense disambigua- tion. In Proceedings of ACL.
  16. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of HLT-NAACL.
  17. Minmin Chen, Yixin Chen, and Kilian Q. Weinberger. 2011. Automatic feature decomposition for single view co-training. In Proceedings of ICML.
  18. Jinho D. Choi, Joel Tetreault, and Amanda Stent. 2015. It depends: Dependency parser comparison using a web-based evaluation tool. In Proceedings of ACL. Open Science Collaboration. 2012. An open, large- scale, collaborative effort to estimate the reproducibil- ity of psychological science. Perspectives on Psycho- logical Science, 7(6):657-660.
  19. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational linguis- tics, 29(4):589-637.
  20. Hal Daumé III. 2007. Frustratingly easy domain adapta- tion. In Proceedings of ACL.
  21. Marie-Catherine De Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Stanford depen- dencies: A cross-linguistic typology. In Proceedings of LREC.
  22. Janez Demšar. 2006. Statistical comparisons of clas- sifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30.
  23. Thomas G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895-1923.
  24. Bradley Efron and Robert J. Tibshirani. 1994. An intro- duction to the bootstrap. CRC press.
  25. Alessio Farcomeni. 2007. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical Methods in Medical Research.
  26. Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at wordvec- tors.org. In Proceedings of the ACL: System Demon- strations.
  27. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001a. Placing search in context: The con- cept revisited. In Proceedings of WWW.
  28. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001b. Placing search in context: The con- cept revisited. In Proceedings of WWW.
  29. Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang, Xian Wu, and Zhong Su. 2009. Domain adapta- tion with latent semantic association for named entity recognition. In Proceedings of HLT-NAACL.
  30. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of ACM SIGKDD.
  31. Ruth Heller, Marina Bogomolov, and Yoav Benjamini. 2014. Deciding whether follow-up studies have repli- cated findings in a preliminary large-scale omics study. Proceedings of the National Academy of Sciences, 111(46):16262-16267.
  32. Thomas Herndon, Michael Ash, and Robert Pollin. 2014. Does high public debt consistently stifle economic growth? a critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2):257-279.
  33. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- tics, 41(4):665-695.
  34. Yosef Hochberg. 1988. A sharper Bonferroni proce- dure for multiple tests of significance. Biometrika, 75(4):800-802.
  35. Sture Holm. 1979. A simple sequentially rejective multi- ple test procedure. Scandinavian Journal of Statistics, 6(2):65-70.
  36. Gerhard Hommel. 1988. A stagewise rejective multi- ple test procedure based on a modified Bonferroni test. Biometrika, 75(2):383-386.
  37. Matthew Honnibal and Mark Johnson. 2015. An im- proved non-monotonic transition system for depen- dency parsing. In Proceedings of EMNLP.
  38. Matthew Honnibal, Yoav Goldberg, and Mark Johnson. 2013. A non-monotonic arc-eager transition system for dependency parsing. In Proceedings of CoNLL.
  39. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation.
  40. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the tenth Machine Translation Summit.
  41. Jeffrey T. Leek and Roger D Peng. 2015. Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Proceedings of the National Academy of Sciences, 112(6):1645-1646.
  42. Omer Levy and Yoav Goldberg. 2014. Dependency- based word embeddings. In Proceedings of ACL.
  43. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Com- positional character models for open vocabulary word representation. In Proceedings of EMNLP.
  44. Thomas M. Loughin. 2004. A systematic compari- son of methods for combining p-values from indepen- dent tests. Computational Statistics & Data Analysis, 47(3):467-485.
  45. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with re- cursive neural networks for morphology. In Proceed- ings of CoNLL.
  46. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beat- rice Santorini. 1993. Building a large annotated cor- pus of English: The Penn Treebank. Computational linguistics, 19(2):313-330.
  47. Edison Marrese-Taylor and Yutaka Matsuo. 2017. Repli- cation issues in syntax-based aspect extraction for opinion mining. In Proceedings of the Student Re- search Workshop at EACL.
  48. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed rep- resentations of words and phrases and their composi- tionality. In Proceedings of NIPS.
  49. George A. Miller and Walter G. Charles. 1991. Contex- tual correlates of semantic similarity. Language and cognitive processes, 6(1):1-28.
  50. Ramal Moonesinghe, Muin J. Khoury, and A. Cecile J. W. Janssens. 2007. Most published research find- ings are false but a little replication goes a long way. PLoS Med, 4(2):e28.
  51. Aurélie Névéol, Cyril Grouin, Kevin Bretonnel Cohen, and Aude Robert. 2016. Replicability of research in biomedical natural language processing: a pilot evalu- ation for a coding task. Proceedings of EMNLP.
  52. Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. The CoNLL 2007 shared task on dependency parsing. In Proceedings of CoNLL.
  53. Diarmuid Ó Séaghdha and Anna Korhonen. 2014. Prob- abilistic distributional semantics. Computational Lin- guistics, 40(3):587-631.
  54. Prasad Patil, Roger D. Peng, and Jeffrey Leek. 2016. A statistical definition for reproducibility and replicabil- ity. bioRxiv.
  55. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word rep- resentation. In Proceedings of EMNLP.
  56. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Notes of the First Workshop on Syntactic Analysis of Non- Canonical Language (SANCL).
  57. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of CoNLL.
  58. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: Com- puting word relatedness using temporal semantic anal- ysis. In Proceedings of WWW.
  59. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627-633.
  60. Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for im- proved word similarity prediction. In Proceedings of CoNLL.
  61. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL.
  62. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast-but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP.
  63. Anders Søgaard. 2013. Estimating effect size across datasets. In Proceedings of HLT-NAACL.
  64. James H. Steiger. 1980. Tests for comparing ele- ments of a correlation matrix. Psychological Bulletin, 87(2):245-251.
  65. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced pro- cessing. Handbook of Natural Language Processing and Machine Translation. Springer.
  66. Frank Wilcoxon. 1945. Individual comparisons by rank- ing methods. Biometrics bulletin, 1(6):80-83.
  67. Dongqiang Yang and David M.W. Powers. Verb similar- ity on the taxonomy of WordNet. In Proceedings of the 3rd International WordNet Conference.
  68. Alexander Yeh. 2000. More accurate tests for the statis- tical significance of result differences. In Proceedings of CoNLL.

Authors
Dror, R., Baumer, G., Bogomolov, M., & Reichart, R.
Affiliations
Technion – Israel Institute of Technology (IL)
ORCID
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2017-12
Volume
5
Pages
471-486
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl_a_00074
OpenAlex
W2963583956
Cited by
63
References
73
Citations by year
2026: 1 · 2025: 5 · 2024: 4 · 2023: 9 · 2022: 9 · 2021: 10 · 2020: 10 · 2019: 14 · 2018: 1
FWCI
4.08
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University)
Funders
Israel Science Foundation (1112/14)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
multiple comparisons, statistical significance testing, NLP evaluation, algorithm comparison, cross-domain sentiment analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review