What's in a p-value in NLP?
Open accessAbstract
In NLP, we need to document that our proposed methods perform significantly better with respect to standard metrics than previous approaches, typically by reporting p-values obtained by rank-or randomization-based tests.We show that significance results following current research standards are unreliable and, in addition, very sensitive to sample size, covariates such as sentence length, as well as to the existence of multiple metrics.We estimate that under the assumption of perfect metrics and unbiased data, we need a significance cut-off at ⇠0.0025 to reduce the risk of false positive results to <5%.Since in practice we often have considerable selection bias and poor metrics, this, however, will not do alone.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Wilcoxon signed-rank test is applicable for most NLP setups.
“The test is actually applicable for most NLP setups and it has been used widely (e.g.(Søgaard et al., 2014;Søgaard, 2013;Yang and Mitchell, 2017)) due to its improved power compared to the sign test variants.”
From The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018) · cites this paper as Søgaard et al., 2014
The Introduction says Wilcoxon’s test is used for comparisons across data sets, and Section 2 says rank-based tests are sometimes used in NLP, especially with few observations. This supports its use in some NLP setups, but not its applicability to most setups or widespread use. Section 2 does not compare its power with sign-test variants; it instead describes rank-based tests as relatively weak compared with bootstrap testing. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 11,966 · see citing works
(2004) · Cited by 9,718 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3× · see citing works
“This is an example of how underpowered studies may introduce false positives (Button et al., 2013).”
BackgroundInfluentialSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
(2005) · Cited by 4,152 · see citing works
Feature-rich part-of-speech tagging with a cyclic dependency network
Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073478 · Cited by 2,889 · see citing works
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788 · see citing works
Domain adaptation with structural correspondence learning
Blitzer, J., McDonald, R., & Pereira, F. (2006) · Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06 · doi:10.3115/1610075.1610094 · Cited by 1,543 · see citing works
Frustratingly Easy Domain AdaptationPreprint
Daumé, H. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0907.1815 · Cited by 1,471 · see citing works
CoNLL-X shared task on multilingual dependency parsing
Buchholz, S., & Marsi, E. (2006) · doi:10.3115/1596276.1596305 · Cited by 971 · see citing works
Parsing with Compositional Vector Grammars
Socher, R., Bauer, J., Manning, C. D., & Ng, A. Y. (2013) · Cited by 830 · see citing works
MaltParser: A language-independent system for data-driven dependency parsing
Nivre, J., Hall, J., Nilsson, J., Chanev, A., Eryiğit, G., Kübler, S., Marinov, S., & Marsi, E. (2007) · Natural Language Engineering · doi:10.1017/s1351324906004505 · Cited by 801 · see citing works
Instance Weighting for Domain Adaptation in NLP
Jiang, J., & Zhai, C. (2007) · Cited by 780 · see citing works
The CoNLL 2007 Shared Task on Dependency Parsing
Nivre, J., Hall, J., Kübler, S., McDonald, R., Nilsson, J., Riedel, S., & Yüret, D. (2007) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · Cited by 667 · see citing works
Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
Clark, J. H., Dyer, C., Lavie, A., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473090 · Cited by 443 · see citing works
Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?
Manning, C. D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19400-9_14 · Cited by 414 · see citing works
Computer Intensive Methods for Testing Hypotheses.
Hollas, E. A., & Noreen, E. W. (1990) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348203 · Cited by 412 · see citing works
Named entity recognition through classifier combination
Florian, R., Ittycheriah, A., Jing, H., & Zhang, T. (2003) · Conference on Computational Natural Language Learning · doi:10.3115/1119176.1119201 · Cited by 401 · see citing works
More accurate tests for the statistical significance of result differences
Yeh, A. (2000) · Proceedings of the 18th conference on Computational linguistics - · doi:10.3115/992730.992783 · Cited by 378 · see citing works
Named entity recognition with a maximum entropy approach
Chieu, H. L., & Ng, H. T. (2003) · doi:10.3115/1119176.1119199 · Cited by 231 · see citing works
(2005) · Cited by 189 · see citing works
(2012) · Cited by 163 · see citing works
Denis, P., & Sagot, B. (2009) · Institutional Repositories DataBase (IRDB) · Cited by 95 · see citing works
Effective Measures of Domain Similarity for Parsing
Plank, B., & van Noord, G. (2011) · Cited by 91 · see citing works
Self-citationUnbounded Dependency Recovery for Parser Evaluation
Rimell, L., Clark, S. C., & Steedman, M. J. (2009) · Cited by 86 · see citing works
From News to Comment: Resources and Benchmarks for Parsing the Language of Web 2.0
Foster, J., Çetinoğlu, Ö., Wagner, J., Roux, J. L., Nivre, J., Hogan, D., & van Genabith, J. (2011) · Arrow@dit (Dublin Institute of Technology) · Cited by 71 · see citing works
Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?
Tsuruoka, Y., Miyao, Y., & Kazama, J. (2011) · Cited by 50 · see citing works
Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation
Schwartz, R., Abend, O., Reichart, R., & Rappoport, A. (2011) · Cited by 42 · see citing works
Parsing with the shortest derivation
Bod, R. (2000) · International Conference on Computational Linguistics · doi:10.3115/990820.990831 · Cited by 41 · see citing works
Cross-Framework Evaluation for Statistical Parsing
Tsarfaty, R., Nivre, J., & Andersson, E. (2012) · Cited by 25 · see citing works
Down-stream effects of tree-to-dependency conversions
Elming, J., Johannsen, A., Klerke, S., Lapponi, E., Alonso, H. M., & Søgaard, A. (2013) · Cited by 23 · see citing works
Self-citationA Multi-Domain Web-Based Algorithm for POS Tagging of Unknown Words
Umansky-Pesin, S., Reichart, R., & Rappoport, A. (2010) · Cited by 23 · see citing works
Learning reliable information for dependency parsing adaptation
Chen, W., Wu, Y., & Isahara, H. (2008) · International Conference on Computational Linguistics · doi:10.3115/1599081.1599096 · Cited by 22 · see citing works
Ranking Human and Machine Summarization Systems
Rankel, P. A., Conroy, J. M., SLUD, E. V., & O’Leary, D. P. (2011) · Cited by 21 · see citing works
Derczynski, L., & Bontcheva, K. (2014) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-4014 · Cited by 9 · see citing works
Flagged first, then by citations
1 of the 39 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Hal Daume III. 2007. Frustratingly easy domain adap- tation. In ACL.
- Roy Schwartz, and Omri Abend, Roi Reichart, and Ari Rappoport. 2011. Neutralizing linguisti- cally problematic annotations in unsupervised de- pendency parsing evaluation. In ACL.
- Anders Søgaard. 2013. Estimating effect size across datasets. In NAACL.
Cited by
Sorted by citations
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing
Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1128 · Cited by 384 · see citing works
Analysis of named entity recognition and linking for tweets
Derczynski, L., Maynard, D., Rizzo, G. F., van Erp, M., Gorrell, G., Raphaël Troncy, A., Petrak, J., & Bontcheva, K. (2014) · Information Processing & Management · doi:10.1016/j.ipm.2014.10.006 · Cited by 340 · see citing works
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks
Faruqui, M., Tsvetkov, Y., Rastogi, P., & Dyer, C. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2506 · Cited by 259 · see citing works
Demographic Factors Improve Classification Performance
Hovy, D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1073 · Cited by 148 · see citing works
Self-citationLearning from Disagreement: A Survey
Uma, A., Fornaciari, T., Hovy, D., Paun, S., Plank, B., & Poesio, M. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12752 · Cited by 154 · see citing works
Self-citationWith Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Accounting for Variance in Machine Learning BenchmarksPreprint
Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., Nichyporuk, B., Szeto, J., Sepah, N., Raff, E., Madan, K., Voleti, V., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03098 · Cited by 115 · see citing works
Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning
Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., & Poesio, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.204 · Cited by 66 · see citing works
Self-citationReplicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
Dror, R., Baumer, G., Bogomolov, M., & Reichart, R. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00074 · Cited by 63 · see citing works
A Game-Theoretic Approach to Word Sense Disambiguation
Tripodi, R., & Pelillo, M. (2016) · Computational Linguistics · doi:10.1162/coli_a_00274 · Cited by 60 · see citing works
Wang, R., & Li, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1405 · Cited by 47 · see citing works
Quantitative proteomics of synaptosome <i>S</i>‐nitrosylation in Alzheimer’s disease
Wijasa, T. S., Sylvester, M., Brocke‐Ahmadinejad, N., Schwartz, S., Santarelli, F., Gieselmann, V., Klockgether, T., Brosseron, F., & Heneka, M. T. (2020) · Journal of Neurochemistry · doi:10.1111/jnc.14870 · Cited by 41 · see citing works
Keystroke dynamics as signal for shallow syntactic parsingPreprint
Plank, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03321 · Cited by 32 · see citing works
Self-citationBERTective: Language Models and Contextual Information for Deception Detection
Fornaciari, T., Bianchi, F., Poesio, M., & Hovy, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.232 · Cited by 20 · see citing works
Self-citationNot All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
Azer, E. S., Khashabi, D., Sabharwal, A., & Roth, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.506 · Cited by 17 · see citing works
Combining Minimally-supervised Methods for Arabic Named Entity Recognition
Althobaiti, M. J., Kruschwitz, U., & Poesio, M. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00136 · Cited by 12 · see citing works
Geolocation with Attention-Based Multitask Learning Models
Fornaciari, T., & Hovy, D. (2019) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/d19-5528 · Cited by 11 · see citing works
Self-citationA Normalizer for UGC in Brazilian Portuguese
Duran, M. S., Nunes, M. D. G. V., & Avanço, L. (2015) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w15-4305 · Cited by 9 · see citing works
Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa
Fornaciari, T., Uma, A., Poesio, M., & Hovy, D. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.12 · Cited by 7 · see citing works
Self-citationIdentifying Linguistic Areas for Geolocation
Fornaciari, T., & Hovy, D. (2019) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/d19-5530 · Cited by 7 · see citing works
Self-citationImproving Lemmatization of Non-Standard Languages with Joint Learning
Manjavacas, E., Kádár, Á., & Kestemont, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1153 · Cited by 7 · see citing works
Attanasio, G., Plaza-del-Arco, F. M., Nozza, D., & Lauscher, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.243 · Cited by 6 · see citing works
Althobaiti, M. J. (2021) · IEEE Access · doi:10.1109/access.2021.3137830 · Cited by 6 · see citing works
Active learning with feature matching for clinical named entity recognition
Le, L., Demartini, G., Zuccon, G., Zhao, G., & Zhang, X. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100015 · Cited by 5 · see citing works
When Sarcasm Hurts: Irony-Aware Models for Abusive Language Detection
Frenda, S., Patti, V., & Rosso, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42448-9_4 · Cited by 4 · see citing works
MilaNLP @ WASSA: Does BERT Feel Sad When You Cry?
Fornaciari, T., Bianchi, F., Nozza, D., & Hovy, D. (2021) · Archivio istituzionale della Ricerca - Bocconi · Cited by 4 · see citing works
Self-citationDense Node Representation for Geolocation
Fornaciari, T., & Hovy, D. (2019) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/d19-5529 · Cited by 3 · see citing works
Self-citationA Two-Sided Discussion of Preregistration of NLP Research
Søgaard, A., Hershcovich, D., & de Lhoneux, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.6 · Cited by 3 · see citing works
Self-citationWe Need to Talk About Reproducibility in NLP Model Comparison
Yan, X., Cao, X., Yang, X., Wang, Y., Wang, R., & Li, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.586 · Cited by 2 · see citing works
Wein, S., Homan, C. M., Aroyo, L. M., & Welty, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.196 · Cited by 2 · see citing works
“We will Reduce Taxes” - Identifying Election Pledges with Language Models
Fornaciari, T., Hovy, D., Naurin, E., Runeson, J., Thomson, R., & Adhikari, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.301 · Cited by 2 · see citing works
Self-citationWord Representations, Tree Models and Syntactic FunctionsPreprint
Šuster, S., Noord, V. G., & Titov, I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.07709 · Cited by 3 · see citing works
Representing Syntax and Composition with Geometric Transformations
Bertolini, L., Weeds, J., Weir, D. J., & Peng, Q. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.296 · Cited by 1 · see citing works
Investigação de métodos de desambiguação lexical de sentidos de verbos do português do Brasil
Cabezudo, M. A. S. (2016) · doi:10.11606/d.55.2016.tde-07012016-112043 · Cited by 1 · see citing works
Not All Claims are Created Equal: Choosing the Right Approach to Assess Your Hypotheses.Preprint
Azer, E. S., Khashabi, D., Sabharwal, A., & Roth, D. (2019) · arXiv (Cornell University)
Switching Contexts: Transportability Measures for NLPPreprint
Marshall, G. C., Thayaparan, M., Osborne, P., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.00823
Representing Syntax and Composition with Geometric TransformationsPreprint
Bertolini, L., Weeds, J., Weir, D. J., & Peng, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01904
Cross-Register Projection for Headline Part of Speech Tagging
Benton, A., Li, H., & Malioutov, I. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5495668
The Impact of Word Embeddings on Neural Dependency Parsing
Adelmann, B., Menzel, W., & Zinsmeister, H. (2021)
- Authors
- Søgaard, A., Johannsen, A., Plank, B., Hovy, D., & Martínez Alonso, H.
- Affiliations
- University of Copenhagen (DK)
- Venue
- Proceedings of the Eighteenth Conference on Computational Natural Language Learning
- Published
- 2014
- Pages
- 1-10
- OpenAlex
- W2111216449
- Cited by
- 39
- References
- 39
- Citations by year
- 2023: 6 · 2022: 1 · 2021: 11 · 2020: 2 · 2019: 7 · 2018: 1 · 2017: 1 · 2016: 4 · 2015: 5 · 2014: 1
- FWCI
- 2.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Computational Natural Language Learning (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- p-values, statistical significance testing, natural language processing, false positives, sample size sensitivity, multiple metrics, selection bias, rank tests, permutation tests, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
