CoChat
Sign Up
Cited by 16

Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP

Belz, Anya, Thomson, Craig, Reiter, Ehud, Mille, Simon·2023·Findings of the Association for Computational Linguistics: ACL 2023
DOI 10.18653/v1/2023.findings-acl.226
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-acl.226

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Human evaluation in NLP for peer review requires effort and expertise, and might be challenging to reproduce.

    “A further drawback of human evaluations in NLP for peer review is that they require effort and expertise, and might be challenging to reproduce (Belz et al., 2023).”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Belz et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. ACM. 2020. Artifact review and badging Version 1.1, August 24, 2020. https://www.acm.org/publications/ policies/artifact-review-and-badging-current.
  2. Anya Belz, Simon Mille, and David M. Howcroft. 2020. Disentangling the properties of human evaluation methods: A classification system to support compa- rability, meta-evaluation and reproducibility testing. In Proceedings of the 13th International Conference on Natural Language Generation, pages 183-194, Dublin, Ireland. Association for Computational Lin- guistics.
  3. Anya Belz, Maja Popovic, and Simon Mille. 2022. Quantified reproducibility assessment of NLP results. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16-28, Dublin, Ireland. Asso- ciation for Computational Linguistics. · DOI
  4. Anya Belz and Ehud Reiter. 2006. Comparing auto- matic and human evaluation of NLG systems. In 11th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 313- 320, Trento, Italy. Association for Computational Linguistics.
  5. Anya Belz, Anastasia Shimorina, Shubham Agarwal, and Ehud Reiter. 2021a. The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Gen- eration, pages 249-258, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  6. Anya Belz, Anastasia Shimorina, Maja Popovic, and Ehud Reiter. 2021b. The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Gen- eration, pages 249-258.
  7. Anya Belz, Craig Thomson, Ehud Reiter, Gavin Aber- crombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Jackie Cheung, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, Tanvi Dinkar, Ondrej Dušek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier González- Corbelle, Dirk Hovy, Manuela Hürlimann, Takumi Ito, John D. Kelleher, Filip Klubicka, Emiel Krah- mer, Huiyuan Lai, Chris van der Lee, Yiru Li, Saad Mahamood, Margot Mieskes, Emiel van Miltenburg, Pablo Mosteiro, Malvina Nissim, Natalie Parde, On- drej Plátek, Verena Rieser, Jie Ruan, Joel Tetreault, Antonio Toral, Xiaojun Wan, Leo Wanner, Lewis Watson, and Diyi Yang. 2023. Missing information, unresponsive authors, experimental flaws: The im- possibility of assessing the reproducibility of previ- ous human evaluations in NLP. In The Fourth Work- shop on Insights from Negative Results in NLP, pages 1-10, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
  8. António Branco, Nicoletta Calzolari, Piek Vossen, Gert- jan Van Noord, Dieter van Uytvanck, João Silva, Luís Gomes, André Moreira, and Willem Elbers. 2020. A shared task of a new, collaborative type to foster re- producibility: A first exercise in the area of language science and technology with REPROLANG2020. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5539-5545, Marseille, France. European Language Resources Association.
  9. Chris Callison-Burch, Cameron Shaw Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the third workshop on statistical ma- chine translation, pages 70-106.
  10. Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic evalu- ation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2748-2760, Florence, Italy. Association for Computational Linguistics. · DOI
  11. Michael Cooper and Matthew Shardlow. 2020. Com- biNMT: An exploration into neural text simplifica- tion models. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 5588-5594, Marseille, France. European Language Resources Association.
  12. Dominic Espinosa, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant. 2010. Further meta- evaluation of broad-coverage surface realization. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 564- 574.
  13. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169-182, Dublin, Ireland. Association for Computational Lin- guistics.
  14. Saad Mahamood. 2021. Reproducing a comparison of hedged and non-hedged NLG texts. In Proceedings of the 14th International Conference on Natural Lan- guage Generation, pages 282-285, Aberdeen, Scot- land, UK. Association for Computational Linguistics.
  15. Saad Mahamood, Ehud Reiter, and Chris Mellish. 2007. A comparison of hedged and non-hedged NLG texts. In Proceedings of the Eleventh European Workshop on Natural Language Generation (ENLG 07), pages 155-158, Saarbrücken, Germany. DFKI GmbH.
  16. Simon Mille, Thiago Castro Ferreira, Anya Belz, and Brian Davis. 2021. Another PASS: A reproduction study of the human evaluation of a football report generation system. In Proceedings of the 14th In- ternational Conference on Natural Language Gen- eration, pages 286-292, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  17. Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text sim- plification models. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 85-91, Vancouver, Canada. Association for Computational Linguistics. · DOI
  18. Maja Popović. 2020. Informative manual evaluation of machine translation output. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5059-5069, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics. · DOI
  19. Maja Popović and Anya Belz. 2021. A reproduction study of an annotation-based human evaluation of MT outputs. In Proceedings of the 14th International Conference on Natural Language Generation, pages 293-300, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  20. Maja Popović, Sheila Castilho, Rudali Huidrom, and Anya Belz. 2022. A reproduction study of an annotation-based human evaluation of MT outputs. In Proceedings of the 15th International Conference on Natural Language Generation, Waterville, Maine, USA and virtual meeting. Association for Computa- tional Linguistics.
  21. Raheel Qader, Khoder Jneid, François Portet, and Cyril Labbé. 2018. Generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation. In Pro- ceedings of the 11th International Conference on Nat- ural Language Generation, pages 254-263, Tilburg University, The Netherlands. Association for Compu- tational Linguistics. · DOI
  22. Christian Richter, Yanran Chen, and Steffen Eger. 2021. TUDA-reproducibility @ ReproGen: Replicability of human evaluation of text-to-text and concept-to-text generation. In Proceedings of the 14th International Conference on Natural Language Generation, pages 301-307, Aberdeen, Scotland, UK. Association for Computational Linguistics.

Cited by

Sorted by citations

Authors
Belz, A., Thomson, C., Reiter, E., & Mille, S.
Venue
Findings of the Association for Computational Linguistics: ACL 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
3676-3687
DOI
10.18653/v1/2023.findings-acl.226
Cited by
16
References
1

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review