We Need to Talk about Standard Splits
Open accessAbstract
It is standard practice in speech & language technology to rank systems according to performance on a test set held out for evaluation.However, few researchers apply statistical tests to determine whether differences in performance are likely to arise by chance, and few examine the stability of system ranking across multiple training-testing splits.We conduct replication and reproduction experiments with nine part-of-speech taggers published between 2000 and 2018, each of which reports state-of-the-art performance on a widely-used "standard split".We fail to reliably reproduce some rankings using randomly generated splits.We suggest that randomly generated splits should be used in system comparison.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Especially when sample sizes are small, significance testing and uncertainty quantification are important steps toward reproducibility.
“Especially when sample sizes are small, significance testing and uncertainty quantification are important steps toward reproducibility.48,85”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [85]
The Introduction (‘Hypothesis testing for system comparison’; page unavailable) argues that performance scores are estimates rather than exact quantities and that hypothesis testing helps avoid false claims of improvement. The Methods (‘Metrics’; page unavailable) reports confidence intervals and a significance test. The paper does not specifically say these steps are especially important when sample sizes are small. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476 · see citing works
Discriminative training methods for hidden Markov models
Collins, M. (2002) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1118693.1118694 · Cited by 1,863 · see citing works
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665 · see citing works
Nonparametric estimates of standard error: The jackknife, the bootstrap and other methods
EFRON, B. (1981) · Biometrika · doi:10.1093/biomet/68.3.589 · Cited by 1,399 · see citing works
TnT: a statistical part-of-speech tagger
Brants, T. (2000) · Proceedings of the sixth conference on Applied natural language processing - · doi:10.3115/974147.974178 · Cited by 1,329 · see citing works
A Maximum Entropy Model for Part-Of-Speech Tagging
Ratnaparkhi, A. (1996) · Cited by 1,268 · see citing works
Some statistical issues in the comparison of speech recognition algorithms
Gillick, L., & Cox, S. J. (2003) · International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1989.266481 · Cited by 636 · see citing works
Reimers, N., & Gurevych, I. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1035 · Cited by 463 · see citing works
Fagerland, M. W., Lydersen, S., & Laake, P. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-91 · Cited by 416 · see citing works
Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?
Manning, C. D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19400-9_14 · Cited by 414 · see citing works
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing
Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1128 · Cited by 384 · see citing works
CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies
Zeman, D., Popel, M., Straka, M., Hajič, J., Nivre, J., Ginter, F., Luotolahti, J., Pyysalo, S., Petrov, S., Potthast, M., et al. (2017) · CoNLL Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies · doi:10.18653/v1/k17-3001 · Cited by 368 · see citing works
TnT - A Statistical Part-of-Speech TaggerPreprint
Brants, T. (2000) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0003055 · Cited by 360 · see citing works
SVMTool: A general POS Tagger Generator Based on Support Vector Machines
Giménez, J., & Màrquez, L. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/52pkqry5zu2h · Cited by 338 · see citing works
Publication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313 · see citing works
Stochastic gradient descent training for L1-regularized log-linear models with cumulative penalty
Tsuruoka, Y., Tsujii, J., & Ananiadou, S. (2009) · doi:10.3115/1687878.1687946 · Cited by 233 · see citing works
sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word EmbeddingsPreprint
Trask, A., Michalak, P., & Liu, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06388 · Cited by 175 · see citing works
CoNLL 2018 Shared Task : Multilingual Parsing from Raw Text to Universal Dependencies
Zeman, D., Hajič, J., Popel, M., Potthast, M., Straka, M., Ginter, F., Nivre, J., & Petrov, S. (2018) · Conference on Computational Natural Language Learning · Cited by 166 · see citing works
Offspring from Reproduction Problems: What Replication Failure Teaches Us
Fokkens, A. S., van Erp, M., Postma, M., Ted Pedersen, A., Vossen, P. T. J. M., & Freire, N. (2013) · Cited by 105 · see citing works
Semi-supervised training for the averaged perceptron POS tagger
Spoustová, J., Hajič, J., Raab, J., & Spousta, M. (2009) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/1609067.1609152 · Cited by 104 · see citing works
Empiricism Is Not a Matter of Faith
Ted Pedersen, A. (2008) · Computational Linguistics · doi:10.1162/coli.2008.34.3.465 · Cited by 96 · see citing works
Denis, P., & Sagot, B. (2009) · Institutional Repositories DataBase (IRDB) · Cited by 95 · see citing works
Reproducibility in Computational Linguistics: Are We Willing to Share?
Wieling, M. B., Rawee, J., & van Noord, G. (2018) · Computational Linguistics · doi:10.1162/coli_a_00330 · Cited by 65 · see citing works
Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
Dror, R., Baumer, G., Bogomolov, M., & Reichart, R. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00074 · Cited by 63 · see citing works
Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?
Tsuruoka, Y., Miyao, Y., & Kazama, J. (2011) · Cited by 50 · see citing works
Dynamic Feature Induction: The Last Gist to the State-of-the-Art
Choi, J. D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1031 · Cited by 42 · see citing works
A Quantitative Study of Data in the NLP community
Mieskes, M. (2017) · doi:10.18653/v1/w17-1603 · Cited by 27 · see citing works
A Challenge Set and Methods for Noun-Verb Ambiguity
El-Kahky, A., Webster, K., Andor, D., & Pitler, E. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1277 · Cited by 18 · see citing works
When data permutations are pathological: the case of neural natural language inference
Schluter, N., & Varab, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1534 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Akbik Alan, Blythe Duncan, and Vollgraf Roland. 2018. Contextual string embedding for sequence labeling. In COLING, pages 1638– 1649.
- Toutanova Kristina, Klein Dan, Manning Christopher D., and Singer Yoram. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL, pages 173– 180.
- Weischedel Ralph, Hovy Eduard, Marcus Mitchell P., Palmer Martha, Belvin Robert, Pradhan Sameer, …, and Nianwen Xue. 2011. OntoNotes: a large training corpus for enhanced processing. In Olive Joseph, Christianson Caitlin, and McCarthy John, editors, Handbook of natural language processing and machine translation, pages 54– 63. Springer, New York.
Cited by
Sorted by citations
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works
“However, one undesirable outcome that has been observed in communities that have adopted the common task framework is a singular focus on optimizing a particular accuracy metric to the exclusion of other scientific and normatively desirable properties of models.(67,85,99) [D5] Lack of computational reproducibility.”
BackgroundZampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.188 · Cited by 389 · see citing works
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233 · see citing works
Toxicity Detection: Does Context Really Matter?
Pavlopoulos, J., Sorensen, J. S., Dixon, L., Thain, N., & Androutsopoulos, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.396 · Cited by 145 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
On the Value of Out-of-Distribution Testing: An Example of Goodhart's LawPreprint
Teney, D., Kafle, K., Shrestha, R., Abbasnejad, E., Kanan, C., & van den Hengel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09241 · Cited by 99 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
Chalkidis, I., Fergadiotis, M., & Androutsopoulos, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.559 · Cited by 89 · see citing works
Accounting for Variance in Machine Learning BenchmarksPreprint
Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., Nichyporuk, B., Szeto, J., Sepah, N., Raff, E., Madan, K., Voleti, V., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03098 · Cited by 115 · see citing works
Antypas, D., Preece, A. D., & Camacho-Collados, J. (2023) · Online Social Networks and Media · doi:10.1016/j.osnem.2023.100242 · Cited by 93 · see citing works
We Need To Talk About Random Splits
Søgaard, A., Ebert, S., Bastings, J., & Filippova, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.156 · Cited by 81 · see citing works
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 74 · see citing works
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 121 · see citing works
SemEval-2021 Task 1: Lexical Complexity Prediction
Shardlow, M., Evans, R., Paetzold, G. H., & Zampieri, M. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.1 · Cited by 59 · see citing works
Sequence Length is a Domain: Length-based Overfitting in Transformer Models
Variš, D., & Bojar, O. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.650 · Cited by 42 · see citing works
A taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41 · see citing works
Teney, D., Abbasnejad, E., Lucey, S., & van den Hengel, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01626 · Cited by 39 · see citing works
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 37 · see citing works
Implicit Discourse Relation Classification: We Need to Talk about Evaluation
Kim, N., Feng, S., Gunasekara, C., & Lastras, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.480 · Cited by 35 · see citing works
Data Leakage in Notebooks: Static Detection and Better Processes
Yang, C., Brower-Sinning, R. A., Lewis, G. A., & Kästner, C. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556918 · Cited by 32 · see citing works
Comparative Evaluation of Label-Agnostic Selection Bias in Multilingual Hate Speech Datasets
Ousidhoum, N. D., Song, Y., & Yeung, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.199 · Cited by 25 · see citing works
(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models’ Performance
Goldman, O., Guriel, D., & Tsarfaty, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.96 · Cited by 23 · see citing works
Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3950378 · Cited by 27 · see citing works
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · arXiv · doi:10.48550/arxiv.1909.03004 · Cited by 22 · see citing works
Interpreting mental state decoding with deep learning models
Thomas, A. W., Ré, C., & Poldrack, R. A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.07.003 · Cited by 22 · see citing works
Voronov, G., Lightheart, R., Frandsen, A., Bargh, B., Haynes, S. E., Spencer, E., Schoenhardt, K. E., Davidson, C., Schaum, A., Macherla, V. R., et al. (2022) · doi:10.1101/2022.10.09.511482 · Cited by 18 · see citing works
Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac297 · Cited by 19 · see citing works
Bassignana, E., & Plank, B. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.7 · Cited by 20 · see citing works
DQI: Measuring Data Quality in NLPPreprint
Mishra, S. R., Arunkumar, A., Sachdeva, B., Bryan, C., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00816 · Cited by 16 · see citing works
Kutuzov, A., & Kuzmenko, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03135 · Cited by 18 · see citing works
Larson, S., Zheng, A., Mahendran, A., Tekriwal, R., Cheung, A., Guldan, E., Leach, K., & Kummerfeld, J. K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.650 · Cited by 14 · see citing works
How Robust are Model Rankings : A Leaderboard Customization Approach for Equitable Evaluation
Mishra, S. R., & Arunkumar, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17599 · Cited by 16 · see citing works
Pampouktsi, P., Avdimiotis, S., Μaragoudakis, M., Avlonitis, M., Samantha, N., Hoogar, P., Ruhago, G. M., & Rono, W. (2022) · Journal of Information System Exploration and Research · doi:10.52465/joiser.v1i1.91 · Cited by 14 · see citing works
On Text-based Personality Computing: Challenges and Future Directions
Fang, Q., Giachanou, A., Bagheri, A., Boeschoten, L., van Kesteren, E., Kamalabad, M. S., & Oberski, D. L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.691 · Cited by 14 · see citing works
What’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 16 · see citing works
ReproducedPapers.org: Openly Teaching and Structuring Machine Learning Reproducibility
Yildiz, B., Hung, H., Krijthe, J. H., Liem, C. C. S., Loog, M., Migut, G., Oliehoek, F. A., Panichella, A., Pawełczak, P., Picek, S., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-76423-4_1 · Cited by 13 · see citing works
It’s about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits
Mu, Y., Bontcheva, K., & Αλέτρας, Ν. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.55 · Cited by 14 · see citing works
We Need to Talk About train-dev-test Splits
van der Goot, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.368 · Cited by 13 · see citing works
HardEval: Focusing on Challenging Tokens to Assess Robustness of NER.
Bernier-Colborne, G., & Langlais, P. (2020) · NPARC · Cited by 12 · see citing works
End-to-End Negation Resolution as Graph Parsing
Kurtz, R., Oepen, S., & Kuhlmann, M. (2020) · International Workshop/Conference on Parsing Technologies · doi:10.18653/v1/2020.iwpt-1.3 · Cited by 14 · see citing works
Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?
Liu, S., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.459 · Cited by 12 · see citing works
Automatic Tuning of Hyperparameters for Neural Networks in Serverless Cloud
Kaplunovich, A., & Yesha, Y. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378280 · Cited by 11 · see citing works
PolyMetriX: an ecosystem for digital polymer chemistry
Kunchapu, S., & Jablonka, K. M. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01823-y · Cited by 10 · see citing works
Modeling Brain Representations of Words' Concreteness in Context Using GPT‐2 and Human Ratings
Bruera, A., Tao, Y., Anderson, A. J., Çokal, D., Haber, J., & Poesio, M. (2023) · Cognitive Science · doi:10.1111/cogs.13388 · Cited by 10 · see citing works
Our Evaluation Metric Needs an Update to Encourage GeneralizationPreprint
Mishra, S. R., Arunkumar, A., Bryan, C., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06898 · Cited by 12 · see citing works
Morphological Segmentation for Seneca
Liu, Z., Jimerson, R., & Prud’hommeaux, E. T. (2021) · Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP) · doi:10.18653/v1/2021.americasnlp-1.10 · Cited by 8 · see citing works
Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation
Liu, Z., & Prud’hommeaux, E. T. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00467 · Cited by 8 · see citing works
Curing the SICK and Other NLI Maladies
Kalouli, A.-L., Hu, H., Webb, A. F., Moss, L. S., & de Paiva, V. (2022) · Computational Linguistics · doi:10.1162/coli_a_00465 · Cited by 8 · see citing works
On the Impact of Temporal Concept Drift on Model Explanations
Zhao, Z., Chrysostomou, G., Bontcheva, K., & Αλέτρας, Ν. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.298 · Cited by 8 · see citing works
Antypas, D., Preece, A. D., & Camacho-Collados, J. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4166108 · Cited by 8 · see citing works
Incorporating Label Dependencies in Multilabel Stance Detection
Ferreira, W. D., & Vlachos, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1665 · Cited by 7 · see citing works
Some Languages Seem Easier to Parse Because Their Treebanks Leak
Søgaard, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.220 · Cited by 6 · see citing works
On the naturalness of hardware descriptions
Lee, J., Nie, P., Li, J. J., & Gligoric, M. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409692 · Cited by 6 · see citing works
Experimental Standards for Deep Learning in Natural Language Processing Research
Ulmer, D., Bassignana, E., Müller-Eberstein, M., Varab, D., Zhang, M., van der Goot, R., Hardmeier, C., & Plank, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.196 · Cited by 6 · see citing works
The Danish Gigaword ProjectPreprint
Derczynski, L., Ciosici, M. R., Baglini, R., Christiansen, M. H., Dalsgaard, J. A., Fusaroli, R., Henrichsen, P. J., Hvingelby, R., Kirkedal, A. S., Kjeldsen, A. S., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03521 · Cited by 6 · see citing works
A pragmatic guide to geoparsing evaluation
Gritta, M., Pilehvar, M. T., & Collier, N. (2019) · Language Resources and Evaluation · doi:10.1007/s10579-019-09475-3 · Cited by 65 · see citing works
Range-Constrained and Stable Federated Charging Scheme for Electricity Vehicles
Zhang, J., Li, S., & Wang, C. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3435001 · Cited by 5 · see citing works
Showing Your Work Doesn’t Always Work
Tang, R., Lee, J., Xin, J., Liu, X., Yu, Y., & Lin, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.246 · Cited by 5 · see citing works
2kenize: Tying Subword Sequences for Chinese Script Conversion
Pranav, A., & Augenstein, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.648 · Cited by 7 · see citing works
Bruera, A., & Poesio, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0291099 · Cited by 4 · see citing works
A randomized prospective study of a hybrid rule- and data-driven virtual patient
Stiff, A., White, M., Fosler‐Lussier, E., Jin, L., Jaffe, E., & Danforth, D. R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000420 · Cited by 4 · see citing works
Is NLP Ready for Standardization?
Aufrant, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.202 · Cited by 4 · see citing works
Derczynski, L., Ciosici, M. R., Baglini, R., Christiansen, M. H., Dalsgaard, J. A., Fusaroli, R., Henrichsen, P. J., Hvingelby, R., Kirkedal, A. S., Kjeldsen, A. S., et al. (2021) · DSpace repository (University of Tartu) · Cited by 4 · see citing works
Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing
Szymański, P., & Gorman, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.172 · Cited by 4 · see citing works
Self-citationDid AI get more negative recently?
Beese, D., Altunbaş, B., Güzeler, G., & Eger, S. (2023) · Royal Society Open Science · doi:10.1098/rsos.221159 · Cited by 4 · see citing works
Çöltekin, Ç. (2020) · Cited by 3 · see citing works
Wecker, H., Friedrich, A., & Adel, H. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.15 · Cited by 3 · see citing works
Explaining Natural Language Processing Classifiers with Occlusion and Language ModelingPreprint
Harbecke, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11889 · Cited by 3 · see citing works
Experimental Standards for Deep Learning in Natural Language Processing ResearchPreprint
Ulmer, D., Bassignana, E., Müller-Eberstein, M., Varab, D., Zhang, M., van der Goot, R., Hardmeier, C., & Plank, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06251 · Cited by 5 · see citing works
We Need to Talk About Reproducibility in NLP Model Comparison
Yan, X., Cao, X., Yang, X., Wang, Y., Wang, R., & Li, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.586 · Cited by 2 · see citing works
Benchmarking Offensive and Abusive Language in Dutch Tweets
Caselli, T., & van der Veen, H. (2023) · doi:10.18653/v1/2023.woah-1.7 · Cited by 2 · see citing works
Wein, S., Homan, C. M., Aroyo, L. M., & Welty, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.196 · Cited by 2 · see citing works
AMR Quality Rating with a Lightweight CNN
Opitz, J. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.27 · Cited by 2 · see citing works
Madureira, B. (2021) · Workshop on Teaching NLP (TeachingNLP) · doi:10.18653/v1/2021.teachingnlp-1.14 · Cited by 2 · see citing works
Automatic Hyperparameter Optimization for Arbitrary Neural Networks in Serverless AWS Cloud
Kaplunovich, A., & Yesha, Y. (2021) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/icics52457.2021.9464618 · Cited by 2 · see citing works
Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis
Seth Kulick, A., Ryant, N., & Santorini, B. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.44 · Cited by 2 · see citing works
Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource SettingPreprint
Lange, L., Adel, H., & Strötgen, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08597 · Cited by 2 · see citing works
Proceedings of the Fifth Workshop on Teaching NLP
(2021) · doi:10.18653/v1/2021.teachingnlp-1 · Cited by 3 · see citing works
Investigating data partitioning strategies for crosslinguistic low-resource ASR evaluation
Liu, Z., Spence, J., & Prud’hommeaux, E. T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.10 · Cited by 2 · see citing works
Stone, S., Schmidt, P., & Birkholz, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053231 · Cited by 1 · see citing works
Demographic Factors as Domains for Adaptation in Linguistic Preprocessing
Morini, S. (2019) · CUNY Academic Works (City University of New York)
Does the Word "Chien" Bark? Representation Learning in Neural Machine Translation Encoders
Campbell, E. (2020) · CUNY Academic Works (City University of New York)
The Impact of Edge Displacement Vaserstein Distance on UD Parsing Performance
Anderson, M., & Gómez‐Rodríguez, C. (2022) · Computational Linguistics · doi:10.1162/coli_a_00440
LITE: Intent-based Task Representation Learning Using Weak Supervision
Otani, N., Gamon, M., Jauhar, S. K., Yang, M., Malireddi, S. R., & Riva, O. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.172
A Survey of MWE Identification Experiments: The Devil is in the Details
Ramisch, C., Walsh, A., Blanchard, T., & Taslimipoor, S. (2023) · Workshop on Multiword Expressions (MWE) · doi:10.18653/v1/2023.mwe-1.15
Can Publicly Available Models Understand NATO Language? A Named Entity Recognition Case Study
D’Ercole, R., Kok, A., Eles, P., Valiyev, G., & Street, M. (2023) · International Conference on Military Communications and Information Systems (ICMCIS) · doi:10.1109/icmcis59922.2023.10253530
Analyzing Inferential Reproducibility
Riezler, S., & Hagmann, M. (2024) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-57065-0_5
Riezler, S., & Hagmann, M. (2024) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-57065-0_2
Evaluating learning trajectories of neural morphology acquisition models
Kodner, J., Khalifa, S., Payne, S., & Liu, Z. (2025) · Linguistics Vanguard · doi:10.1515/lingvan-2024-0215
Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair
Han, W., Kwak, Y., Yu, M., Kim, K., Lee, Y., Moon, H., & Paek, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773167
Bayesian Model Comparison Based on Cross-Validated Estimation of $$\text {F}_1$$ Measure
Xue, Y., Wang, Y., Cao, X., Yang, X., & Li, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4987-0_19
Unsupervised multilingual models of speech representation, an approach inspired by cognitive science
de Seyssel, M. (2023) · theses.fr (ABES) · doi:10.70675/2974e23cz54f3z4282z8000z612287446e7f
ABWS: The Arabic Boundary-aware Word Segmentation Benchmarkfor Reproducible Evaluation
Alshuhayeb, H., & Bidgoli, B. M. (2026) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.66834/9a7y0c13
Pereira, V. B. S., Nunes, R. O., & Balreira, D. G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39895-6_35
From Extracted Numbers to Comparable Evidence: Protocol-Qualified Scientific Result Records
Siravuri, H. V. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3847124
Sammo, N. S., Lura, S. A., Joy, M. J. R., Nowrozy, R., & Bevinakoppa, S. (2026) · Electronics · doi:10.3390/electronics15204723
PII-API-TR-v1: A Turkish Multi-Label API Payload Dataset for PII Detection
Gümüş, R. B., Çakır, D., & Kalpakoğlu, E. C. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11728167
- Authors
- Gorman, K., & Bedrick, S.
- Affiliations
- City University of New York (US)
- Oregon Health & Science University (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 2786-2791
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC10287171
- OpenAlex
- W2951243568
- Cited by
- 99
- References
- 30
- Citations by year
- 2026: 8 · 2025: 2 · 2024: 4 · 2023: 18 · 2022: 22 · 2021: 16 · 2020: 23 · 2019: 6
- FWCI
- 6.50
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health
- National Institute on Deafness and Other Communication Disorders (R01DC015999)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- part-of-speech tagging, system ranking, statistical significance, replication, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
