The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Evaluating commit message generation: to BLEU or not to BLEU?
Open accessAbstract
Commit messages play an important role in several software engineering tasks such as program comprehension and understanding program evolution. However, programmers neglect to write good commit messages. Hence, several Commit Message Generation (CMG) tools have been proposed. We observe that the recent state of the art CMG tools use simple and easy to compute automated evaluation metrics such as BLEU4 or its variants. The advances in the field of Machine Translation (MT) indicate several weaknesses of BLEU4 and its variants. They also propose several other metrics for evaluating Natural Language Generation (NLG) tools. In this work, we discuss the suitability of various MT metrics for the CMG task. Based on the insights from our experiments, we propose a new variant specifically for evaluating the CMG task. We re-evaluate the state of the art CMG tools on our new metric. We believe that our work fixes an important gap that exists in the understanding of evaluation metrics for CMG research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“BLEU [50] is the most widely-used, similarity-based measure for code summarization [57] and commit log generation [16].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [16]
Citations
Cites
A Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386
A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU
Chen, B., & Cherry, C. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3346 · Cited by 191
A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes
Loyola, P., Marrese-Taylor, E., & Matsuo, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2045 · Cited by 128
Flagged first, then by citations
Research works printed but not linked: 17
- Satanjeev Banerjee and Alon Lavie . 2005 . METEOR: An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65--72 . Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65--72.
- Asli Celikyilmaz , Elizabeth Clark , and Jianfeng Gao . 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799 ( 2020 ). Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799 (2020). · arXiv
- Shweta Chauhan , Philemon Daniel , Archita Mishra , and Abhay Kumar . 2021. AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages. IETE Journal of Research ( 2021 ), 1--12. Shweta Chauhan, Philemon Daniel, Archita Mishra, and Abhay Kumar. 2021. AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages. IETE Journal of Research (2021), 1--12.
- Michael Denkowski and Alon Lavie . 2010 . METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support For Five Target Languages . In Proceedings of the ACL 2010 Joint Workshop on Statistical Machine Translation and Metrics MATR. Michael Denkowski and Alon Lavie. 2010. METEOR-NEXT and the METEOR Paraphrase Tables: Improved Evaluation Support For Five Target Languages. In Proceedings of the ACL 2010 Joint Workshop on Statistical Machine Translation and Metrics MATR.
- Lifeng Han . 2016. Machine translation evaluation resources and methods: A survey. arXiv preprint arXiv:1605.04515 ( 2016 ). Lifeng Han. 2016. Machine translation evaluation resources and methods: A survey. arXiv preprint arXiv:1605.04515 (2016). · arXiv
- 10.1109/ASE.2017.8115626 · DOI
- 10.3115/1557769.1557821 · DOI · Paper page
- Chin-Yew Lin . 2004 . Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
- 10.1145/3238147.3238190 · DOI
- Minh-Thang Luong , Hieu Pham , and Christopher D Manning . 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 ( 2015 ). Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015). · arXiv
- 10.18653/v1/2020.acl-main.448 · DOI
- 10.3115/1073083.1073135 · DOI · Paper page
- Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311--318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311--318.
- Ananya B Sai , Akash Kumar Mohankumar, and Mitesh M Khapra . 2020 . A survey of evaluation metrics used for NLG systems. arXiv preprint arXiv:2008.12009 (2020). Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2020. A survey of evaluation metrics used for NLG systems. arXiv preprint arXiv:2008.12009 (2020). · arXiv
- Matthew Snover , Bonnie Dorr , Richard Schwartz , Linnea Micciulla , and John Makhoul . 2006 . A study of translation edit rate with targeted human annotation . In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers. 223--231 . Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers. 223--231.
- Wei Tao , Yanlin Wang , Ensheng Shi , Lun Du , Hongyu Zhang , Dongmei Zhang , and Wenqiang Zhang . 2021. On the Evaluation of Commit Message Generation Models: An Experimental Study. arXiv preprint arXiv:2107.05373 ( 2021 ). Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the Evaluation of Commit Message Generation Models: An Experimental Study. arXiv preprint arXiv:2107.05373 (2021). · arXiv
- Jerrold H Zar . 2005. Spearman rank correlation. Encyclopedia of biostatistics 7 ( 2005 ). Jerrold H Zar. 2005. Spearman rank correlation. Encyclopedia of biostatistics 7 (2005).
Cited by
Sorted by citations
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639183 · Cited by 112
“However, for up to 2% of the randomly chosen samples in each experiment, we didn’t get good results; either the prompt didn’t fit into the model’s 5 Please see Section 7 for the rationale. window, or the model mysteriously generated an empty string.”
BackgroundMethod
Metadata
- Authors
- Dey, S., Vinayakarao, V., Gupta, M., & Dechu, S.
- Affiliations
- Chennai Mathematical Institute (IN)
- IBM Research - India (IN)
- Venue
- Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 31-35
- OpenAlex
- W4224313199
- Cited by
- 9
- References
- 3
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 6 · 2022: 1
- FWCI
- 1.79
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- commit message generation, evaluation metrics, BLEU, natural language generation evaluation, version control, program comprehension, software evolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
