CoChat
Sign Up
Cited by 86Scientific Computing and Data ManagementNatural Language Processing TechniquesTopic Modeling

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

A Systematic Review of Reproducibility Research in Natural Language Processing

Open access
Belz, Anya, Agarwal, Shubham, Shimorina, Anastasia, Reiter, Ehud·2021·Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
DOI 10.18653/v1/2021.eacl-main.29
Open this paper in CoChat — free accountOpen PDF

Abstract

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results.The past few years have seen an impressive range of new initiatives, events and active research in the area.However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging.With this focused contribution, we aim to provide a wideangle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.eacl-main.29

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Systematic reviews of NLP reproduction studies report that only a small fraction of attempted reproductions recover the original scores.

    “Systematic reviews of NLP reproduction studies report that only a small fraction of attempted reproductions recover the original scores (Belz et al., 2021)”

    From ReproAgent: Contract-Guided Paper-to-Code Reproduction (2026) · cites this paper as Belz et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Mohamed Abdellatif and Ahmed Elgammal. 2020. ULMFiT replication. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5579-5587, Marseille, France. European Lan- guage Resources Association.
  2. Cristina Arhiliuc, Jelena Mitrović, and Michael Gran- itzer. 2020. Language proficiency scoring. In Pro- ceedings of The 12th Language Resources and Eval- uation Conference, pages 5624-5630, Marseille, France. European Language Resources Association.
  3. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 789-798, Melbourne, Australia. As- sociation for Computational Linguistics. Association for Computing Machinery. 2020. Artifact review and badging Ver- sion 1.1, August 24, 2020. https: //www.acm.org/publications/policies/ artifact-review-and-badging-current. · DOI
  4. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759-760.
  5. Monya Baker. 2016. Reproducibility crisis. Nature, 533(26):353-66.
  6. Yves Bestgen. 2020. Reproducing monolingual, multi- lingual and cross-lingual CEFR predictions. In Pro- ceedings of The 12th Language Resources and Eval- uation Conference, pages 5595-5602, Marseille, France. European Language Resources Association.
  7. Daniel M. Bikel. 2004. Intricacies of collins' parsing model. Computational Linguistics, 30(4):479-511. · DOI
  8. Bernd Bohnet, Ryan McDonald, Gonc ¸alo Simões, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018. Morphosyntactic tagging with a meta- BiLSTM model over context sensitive token encod- ings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2642-2652, Melbourne, Australia. Association for Computational Linguis- tics. · DOI
  9. S.R.K. Branavan, David Silver, and Regina Barzilay. 2011. Learning to win by reading manuals in a monte-Carlo framework. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies, pages 268-277, Portland, Oregon, USA. Associa- tion for Computational Linguistics.
  10. António Branco, Nicoletta Calzolari, Piek Vossen, Gertjan Van Noord, Dieter van Uytvanck, João Silva, Luís Gomes, André Moreira, and Willem Elbers. 2020. A shared task of a new, collaborative type to foster reproducibility: A first exercise in the area of language science and technology with RE- PROLANG2020. In Proceedings of The 12th Lan- guage Resources and Evaluation Conference, pages 5539-5545, Marseille, France. European Language Resources Association.
  11. Ana Brassard, Tin Kuculo, Filip Boltužić, and Jan Šnajder. 2018. TakeLab at SemEval-2018 task12: Argument reasoning comprehension with skip-thought vectors. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1133-1136, New Orleans, Louisiana. Associ- ation for Computational Linguistics. · DOI
  12. Andrew Caines and Paula Buttery. 2020. RE- PROLANG 2020: Automatic proficiency scoring of Czech, English, German, Italian, and Spanish learner essays. In Proceedings of The 12th Lan- guage Resources and Evaluation Conference, pages 5614-5623, Marseille, France. European Language Resources Association.
  13. HongSeok Choi and Hyunju Lee. 2018. GIST at SemEval-2018 task 12: A network transferring infer- ence knowledge to argument reasoning comprehen- sion task. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 773-777, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI
  14. Maximin Coavoux and Benoît Crabbé. 2016. Neural greedy constituent parsing with dynamic oracles. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 172-182, Berlin, Germany. As- sociation for Computational Linguistics. · DOI
  15. K Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany J Callahan, Orin Hargraves, Foster Goss, Nancy Ide, Aurélie Névéol, Cyril Grouin, and Lawrence E Hunter. 2018. Three dimensions of reproducibility in natural language processing. In LREC... International Conference on Language Re- sources & Evaluation:[proceedings]. International Conference on Language Resources and Evaluation, volume 2018, page 156. NIH Public Access.
  16. Michael Collins. 1999. Head-driven statistical models for natural language parsing. Ph.D. thesis.
  17. Michael Cooper and Matthew Shardlow. 2020. Com- biNMT: An exploration into neural text simplifica- tion models. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5588- 5594, Marseille, France. European Language Re- sources Association.
  18. Matt Crane. 2018. Questionable answers in question answering research: Reproducibility and variability of published results. Transactions of the Association for Computational Linguistics, 6:241-252. · DOI
  19. Daniel Dakota and Sandra Kübler. 2017. Towards replicability in parsing. In Proceedings of the In- ternational Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 185-194, Varna, Bulgaria. INCOMA Ltd. · DOI
  20. Chris Drummond. 2009. Replicability is not repro- ducibility: nor is it good science. Presented at 4th Workshop on Evaluation Methods for Machine Learning held at ICML'09.
  21. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Off- spring from reproduction problems: What replica- tion failure teaches us. In Proceedings of the 51st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 1691-1701, Sofia, Bulgaria. Association for Compu- tational Linguistics.
  22. Paula Fortuna, Juan Soler-Company, and Sérgio Nunes. 2019. Stop PropagHate at SemEval-2019 tasks 5 and 6: Are abusive language classification results reproducible? In Proceedings of the 13th Inter- national Workshop on Semantic Evaluation, pages 745-752, Minneapolis, Minnesota, USA. Associa- tion for Computational Linguistics. · DOI
  23. Nuno Freire, José Borbinha, and Pável Calado. 2012. An approach for named entity recognition in poorly structured data. In Extended Semantic Web Confer- ence, pages 718-732. Springer.
  24. Qiaozi Gao, Malcolm Doering, Shaohua Yang, and Joyce Chai. 2016. Physical causality of action verbs in grounded language understanding. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 1814-1824, Berlin, Germany. Associa- tion for Computational Linguistics. · DOI
  25. Nicolas Garneau, Mathieu Godbout, David Beau- chemin, Audrey Durand, and Luc Lamontagne. 2020. A robust self-learning method for fully un- supervised cross-lingual mappings of word embed- dings: Making the method robustly reproducible as well. In Proceedings of The 12th Language Re- sources and Evaluation Conference, pages 5546- 5554, Marseille, France. European Language Re- sources Association.
  26. Daniel Gildea. 2001. Corpus variation and parser per- formance. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Process- ing.
  27. Yulan He, Chenghua Lin, and Harith Alani. 2011. Automatically extracting polarity-bearing topics for cross-domain sentiment classification. In Proceed- ings of the 49th Annual Meeting of the Associa- tion for Computational Linguistics: Human Lan- guage Technologies, pages 123-131, Portland, Ore- gon, USA. Association for Computational Linguis- tics.
  28. Tobias Horsmann and Torsten Zesch. 2017. Do LSTMs really work so well for PoS tagging? -a replication study. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 727-736, Copenhagen, Denmark. Association for Computational Linguistics. · DOI
  29. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics. · DOI · Paper page
  30. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised def- initions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169-182, Dublin, Ireland. Association for Computa- tional Linguistics.
  31. Phu Mon Htut, Kyunghyun Cho, and Samuel Bowman. 2018a. Grammar induction with neural language models: An unusual replication. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 4998-5003, Brus- sels, Belgium. Association for Computational Lin- guistics. · DOI
  32. Phu Mon Htut, Kyunghyun Cho, and Samuel Bowman. 2018b. Grammar induction with neural language models: An unusual replication. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks for NLP, pages 371-373, Brussels, Belgium. Association for Computational Linguistics. · DOI
  33. Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing deep neu- ral networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2410-2420, Berlin, Germany. Association for Com- putational Linguistics. · DOI
  34. Eva Huber and C ¸agrı C ¸öltekin. 2020. Reproduction and replication: A case study with automatic es- say scoring. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5603- 5613, Marseille, France. European Language Re- sources Association.
  35. Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional seman- tics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 590-599, Port- land, Oregon, USA. Association for Computational Linguistics.
  36. Margot Mieskes, Karën Fort, Aurélie Névéol, Cyril Grouin, and Kevin Cohen. 2019. Community per- spective on replicability in natural language process- ing. In Proceedings of the International Conference on Recent Advances in Natural Language Process- ing (RANLP 2019), pages 768-775, Varna, Bulgaria. · DOI
  37. Alice Millour, Karën Fort, and Pierre Magistry. 2020. Répliquer et étendre pour l'alsacien "étiquetage en parties du discours de langues peu dotées par spécialisation des plongements lexicaux" (replicat- ing and extending for Alsatian : "POS tagging for low-resource languages by adapting word embed- dings"). In Actes de la 6e conférence conjointe Journées d' Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Au- tomatique des Langues (R ÉCITAL, 22e édition). 2e atelier Éthique et TRaitemeNt Automatique des Langues (ETeRNAL), pages 29-37, Nancy, France.
  38. Andrew Moore and Paul Rayson. 2018. Bringing repli- cation and reproduction together with generalisabil- ity in NLP: Three reproduction studies for target dependent sentiment analysis. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1132-1144, Santa Fe, New Mex- ico, USA. Association for Computational Linguis- tics.
  39. Preslav Nakov and Hwee Tou Ng. 2011. Translat- ing from morphologically complex languages: A paraphrase-based approach. In Proceedings of the 49th Annual Meeting of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 1298-1307, Portland, Oregon, USA. As- sociation for Computational Linguistics.
  40. Garrett Nicolai and Grzegorz Kondrak. 2016. Lever- aging inflection tables for stemming and lemmatiza- tion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1138-1147, Berlin, Ger- many. Association for Computational Linguistics. · DOI
  41. Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text sim- plification models. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 85-91, Vancouver, Canada. Association for Computational Linguistics. · DOI
  42. Timothy Niven and Hung-Yu Kao. 2018. NLITrans at SemEval-2018 task 12: Transfer of semantic knowledge for argument comprehension. In Pro- ceedings of The 12th International Workshop on Se- mantic Evaluation, pages 1099-1103, New Orleans, Louisiana. Association for Computational Linguis- tics. · DOI
  43. Babatunde Kazeem Olorisade, Pearl Brereton, and Pe- ter Andras. 2017. Reproducibility of studies on text mining for citation screening in systematic reviews: evaluation and checklist. Journal of biomedical in- formatics, 73:1-13.
  44. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NeurIPS-W.
  45. Ted Pedersen. 2008. Empiricism is not a matter of faith. Computational Linguistics, 34(3):465-470.
  46. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidi- rectional long short-term memory models and auxil- iary loss. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412-418, Berlin, Germany. Association for Computational Linguis- tics. · DOI
  47. Kamil Pluciński, Mateusz Lango, and Michał Zimniewicz. 2020. A closer look on unsuper- vised cross-lingual word embeddings mapping. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5555-5562, Marseille, France. European Language Resources Association.
  48. Kyeongmin Rim, Jingxuan Tu, Kelley Lynch, and James Pustejovsky. 2020. Reproducing neural en- semble classifier for semantic relation extraction in- Scientific papers. In Proceedings of The 12th Lan- guage Resources and Evaluation Conference, pages 5569-5578, Marseille, France. European Language Resources Association.
  49. João Rodrigues, Ruben Branco, João Silva, and António Branco. 2020. Reproduction and revival of the argument reasoning comprehension task. In Pro- ceedings of The 12th Language Resources and Eval- uation Conference, pages 5055-5064, Marseille, France. European Language Resources Association.
  50. Jonathan Rotsztejn, Nora Hollenstein, and Ce Zhang. 2018. ETH-DS3Lab at SemEval-2018 task 7: Ef- fectively combining recurrent and convolutional neu- ral networks for relation classification and extrac- tion. In Proceedings of The 12th International Work- shop on Semantic Evaluation, pages 689-696, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  51. Christina Sauper, Aria Haghighi, and Regina Barzilay. 2011. Content models with attitude. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, pages 350-358, Portland, Oregon, USA. Association for Computational Linguistics.
  52. Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville. 2018. Neural language modeling by jointly learning syntax and lexicon. In Interna- tional Conference on Learning Representations.
  53. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective lstms for target-dependent senti- ment classification. In Proceedings of COLING 2016, the 26th International Conference on Compu- tational Linguistics: Technical Papers, pages 3298- 3307.
  54. Junfeng Tian, Man Lan, and Yuanbin Wu. 2018. ECNU at SemEval-2018 task 12: An end-to-end attention-based neural network for the argument rea- soning comprehension task. In Proceedings of The 12th International Workshop on Semantic Evalua- tion, pages 1094-1098, New Orleans, Louisiana. As- sociation for Computational Linguistics. · DOI
  55. Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Learning semantically and additively compositional distributional representations. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1277-1287, Berlin, Germany. Association for Com- putational Linguistics. · DOI
  56. Sowmya Vajjala and Taraka Rama. 2018. Experiments with universal CEFR classification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 147-153, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  57. Bo Wang, Maria Liakata, Arkaitz Zubiaga, and Rob Procter. 2017. Tdparse: Multi-target-specific senti- ment recognition on twitter. In Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics: Volume 1, Long Papers, pages 483-493.
  58. Martijn Wieling, Josine Rawee, and Gertjan van Noord. 2018. Squib: Reproducibility in computational lin- guistics: Are we willing to share? Computational Linguistics, 44(4):641-649. · DOI
  59. Meiqian Zhao, Chunhua Liu, Lu Liu, Yan Zhao, and Dong Yu. 2018. BLCU NLP at SemEval-2018 task 12: An ensemble model for argument reasoning based on hierarchical attention. In Proceedings of The 12th International Workshop on Semantic Eval- uation, pages 1104-1108, New Orleans, Louisiana. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Metadata

Authors
Belz, A., Agarwal, S., Shimorina, A., & Reiter, E.
Affiliations
  • Heriot-Watt University (GB)
  • Laboratoire Lorrain de Recherche en Informatique et ses Applications (FR)
  • University of Aberdeen (GB)
ORCID
Venue
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Publisher
Association for Computational Linguistics
Published
2021
Pages
381-393
DOI
10.18653/v1/2021.eacl-main.29
OpenAlex
W3139394221
Cited by
86
References
2
Citations by year
2026: 6 · 2025: 7 · 2024: 30 · 2023: 21 · 2022: 14 · 2021: 8
FWCI
3.24
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Dublin City University Open Access Institutional Repository (Dublin City University) (submittedVersion)
  • Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
reproducibility, replication crisis, natural language processing, research reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review