Correlating Automated and Human Evaluation of Code Documentation Generation Quality
Open accessAbstract
Automatic code documentation generation has been a crucial task in the field of software engineering. It not only relieves developers from writing code documentation but also helps them to understand programs better. Specifically, deep-learning-based techniques that leverage large-scale source code corpora have been widely used in code documentation generation. These works tend to use automatic metrics (such as BLEU, METEOR, ROUGE, CIDEr, and SPICE) to evaluate different models. These metrics compare generated documentation to reference texts by measuring the overlapping words. Unfortunately, there is no evidence demonstrating the correlation between these metrics and human judgment. We conduct experiments on two popular code documentation generation tasks, code comment generation and commit message generation, to investigate the presence or absence of correlations between these metrics and human judgments. For each task, we replicate three state-of-the-art approaches and the generated documentation is evaluated automatically in terms of BLEU, METEOR, ROUGE-L, CIDEr, and SPICE. We also ask 24 participants to rate the generated documentation considering three aspects (i.e., language, content, and effectiveness). Each participant is given Java methods or commit diffs along with the target documentation to be rated. The results show that the ranking of generated documentation from automatic metrics is different from that evaluated by human annotators. Thus, these automatic metrics are not reliable enough to replace human evaluation for code documentation generation tasks. In addition, METEOR shows the strongest correlation (with moderate Pearson correlation r about 0.7) to human evaluation metrics. However, it is still much lower than the correlation observed between different annotators (with a high Pearson correlation r about 0.8) and correlations that are reported in the literature for other tasks (e.g., Neural Machine Translation [ 39 ]). Our study points to the need to develop specialized automated evaluation metrics that can correlate more closely to human evaluation metrics for code generation tasks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
InCoder-1b has relatively fewer parameters, and it is still challenging for it to understand and capture the content and associations of multiple examples in the input.
“The reason may be that InCoder-1b has relatively fewer parameters, and it is still challenging for it to understand and capture the content and associations of multiple examples in the input [30].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [30]
Traditional NLP metrics such as ROUGE, METEOR, and BLEU are known to poorly correlate with human evaluations of natural language generations.
“Traditional NLP metrics such as ROUGE, METEOR, and BLEU—which simply count n-gram overlap between generated and reference text—are known to poorly correlate with human evaluations of natural language generations83–85.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [84]
Cites
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
Applied Statistics for the Behavioral Sciences
Witz, K. G., Hinkle, D. E., Wiersma, W., & Jurs, S. G. (1990) · Journal of Educational Statistics · doi:10.2307/1164825 · Cited by 2,927 · see citing works
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,879 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Noise Reduction in Speech Processing
Cohen, I., Huang, Y. A., Chen, J., & Benesty, J. (2009) · Springer topics in signal processing · doi:10.1007/978-3-642-00296-0 · Cited by 1,466 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
Summarizing Source Code using a Neural Attention Model
Iyer, S. V., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1195 · Cited by 716 · see citing works
Murray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2005) · Meeting of the Association for Computational Linguistics · Cited by 716 · see citing works
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. G. (2018) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3196321.3196334 · Cited by 705 · see citing works
Self-citationTowards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · see citing works
On the Use of Automated Text Summarization Techniques for Summarizing Source Code
Haiduc, S., Aponte, J., Moreno, L., & Marcus, A. (2010) · 2010 17th Working Conference on Reverse Engineering · doi:10.1109/wcre.2010.13 · Cited by 408 · see citing works
Improving automatic source code summarization via deep reinforcement learning
Wan, Y., Zhao, Z., Yang, M., Xu, G., Ying, H., Wu, J., & Yu, P. S. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238206 · Cited by 392 · see citing works
How software engineers use documentation: the state of the practice
Lethbridge, T. C., Singer, J., & Forward, A. (2003) · IEEE Software · doi:10.1109/ms.2003.1241364 · Cited by 340 · see citing works
Automatic generation of natural language summaries for Java classes
Moreno, L., Aponte, J., Sridhara, G., Marcus, A., Pollock, L., & Vijay‐Shanker, K. (2013) · International Conference on Program Comprehension · doi:10.1109/icpc.2013.6613830 · Cited by 340 · see citing works
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326 · see citing works
Self-citationImproved Code Summarization via a Graph Neural Network
LeClair, A., Haque, S., Wu, L., & McMillan, C. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389268 · Cited by 322 · see citing works
A Neural Model for Generating Natural Language Summaries of Program Subroutines
LeClair, A., Jiang, S., & McMillan, C. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00087 · Cited by 305 · see citing works
Supporting program comprehension with source code summarization
Haiduc, S., Aponte, J., & Marcus, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2 · doi:10.1145/1810295.1810335 · Cited by 291 · see citing works
Deep code comment generation with hybrid lexical and syntactical information
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. G. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09730-9 · Cited by 279 · see citing works
Self-citationRetrieval-based neural source code summarization
Zhang, J., Wang, X., Zhang, H., Sun, H., & Liu, X. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380383 · Cited by 275 · see citing works
The relevance of software documentation, tools and technologies: a survey
Forward, A., & Lethbridge, T. C. (2002) · Proceedings of the 2002 ACM symposium on Document engineering · doi:10.1145/585058.585065 · Cited by 268 · see citing works
Automatically generating commit messages from diffs using neural machine translation
Jiang, S., Armaly, A., & McMillan, C. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115626 · Cited by 267 · see citing works
Neural-machine-translation-based commit message generation: how far are we?
Liu, Z., Xia, X., Hassan, A. E., Lo, D., Xing, Z., & Wang, X. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238190 · Cited by 250 · see citing works
Self-citationBoa: A Language and Infrastructure for Analyzing Ultra-Large-Scale Software Repositories
Dyer, R., Nguyen, H. A., Rajan, H., & Nguyen, T. N. (2013) · Cited by 246 · see citing works
Automatically documenting program changes
Buse, R. P. L., & Weimer, W. R. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859005 · Cited by 219 · see citing works
Automatic documentation generation via source code summarization of method context
McBurney, P. W., & McMillan, C. (2014) · International Conference on Program Comprehension · doi:10.1145/2597008.2597149 · Cited by 217 · see citing works
On Automatically Generating Commit Messages via Summarization of Source Code Changes
Cortes-Coy, L. F., Linares‐Vásquez, M., Aponte, J., & Poshyvanyk, D. (2014) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2014.14 · Cited by 165 · see citing works
Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse
Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K. K., Koehn, P., Palmer, M. S., & Schneider, N. (2013) · The Association for Computational Linguistics · Cited by 133 · see citing works
A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes
Loyola, P., Marrese-Taylor, E., & Matsuo, Y. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2045 · Cited by 128 · see citing works
Evaluating Evaluation Methods for Generation in the Presence of Variation
Stent, A., Marge, M., & Singhai, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30586-6_38 · Cited by 126 · see citing works
ChangeScribe: A Tool for Automatically Generating Commit Messages
Linares‐Vásquez, M., Cortes-Coy, L. F., Aponte, J., & Poshyvanyk, D. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.229 · Cited by 96 · see citing works
Statistical Methods for Rates and Proportions
Plackett, R. L., & Fleiss, J. L. (1975) · International Statistical Review · doi:10.2307/1402915 · Cited by 92 · see citing works
A Human Study of Comprehension and Code Summarization
Stapleton, S., Gambhir, Y., LeClair, A., Eberhart, Z., Weimer, W. R., Leach, K., & Huang, Y. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389258 · Cited by 81 · see citing works
A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques
Song, X., Sun, H.-L., Wang, X., & Yan, J. (2019) · IEEE Access · doi:10.1109/access.2019.2931579 · Cited by 68 · see citing works
Generating Commit Messages from Diffs using Pointer-Generator Network
Liu, Q., Liu, Z., Zhu, H., Fan, H., Du, B., & Qian, Y. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00056 · Cited by 68 · see citing works
Why My Code Summarization Model Does Not Work
Chen, Q., Xia, X., Hu, H., Lo, D., & Li, S. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3434280 · Cited by 58 · see citing works
Self-citationMining Version Control System for Automatically Generating Commit Comment
Huang, Y., Zheng, Q., Chen, X., Xiong, Y., Liu, Z., & Luo, X. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.56 · Cited by 57 · see citing works
JSummarizer: An automatic generator of natural language summaries for Java classes
Moreno, L., Marcus, A., Pollock, L., & Vijay‐Shanker, K. (2013) · International Conference on Program Comprehension · doi:10.1109/icpc.2013.6613855 · Cited by 55 · see citing works
Retrieve and Refine: Exemplar-Based Neural Comment Generation
Wei, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00152 · Cited by 51 · see citing works
Gros, D., Sezhiyan, H., Dévanbu, P., & Zhou, Y. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416546 · Cited by 42 · see citing works
Content Aware Source Code Change Description Generation
Loyola, P., Marrese-Taylor, E., Balazs, J., Matsuo, Y., & Satoh, F. (2018) · International Conference on Natural Language Generation · doi:10.18653/v1/w18-6513 · Cited by 26 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations
Tony, C., Mutas, M., Ferreyra, N. E. D., & Scandariato, R. (2023) · 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00084 · Cited by 61 · see citing works
ExploitGen: Template-augmented exploit code generation based on CodeBERT
Yang, G., Zhou, Y., Chen, X., Zhang, X., Han, T., & Chen, T. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111577 · Cited by 50 · see citing works
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
“Table 16 shows the best-performing models from small pre-trained models, LLM-ICLs and LLM-PEFTs selected to answer this research question.”
Self-citationEvaluating Code Summarization Techniques: A New Metric and an Empirical Characterization
Mastropaolo, A., Ciniselli, M., Penta, M. D., & Bavota, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639174 · Cited by 40 · see citing works
Who evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators
Liguori, P., Improta, C., Natella, R., Čukić, B., & Cotroneo, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120073 · Cited by 31 · see citing works
Huang, Y., Wang, W., Zhou, J., Zhang, L., Lin, J., Liu, H., Hu, X., Zhou, Z., & Dong, W. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104152 · Cited by 20 · see citing works
Self-citationTransformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 23 · see citing works
AttSum: A Deep Attention-Based Summarization Model for Bug Report Title Generation
Ma, X., Keung, J., Yu, X., Zou, H., Zhang, J., & Li, Y. (2023) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3236404 · Cited by 17 · see citing works
KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation
Tao, W., Zhou, Y., Wang, Y., Zhang, H., Wang, H., & Zhang, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643675 · Cited by 16 · see citing works
Self-citationHu, X. X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 23 · see citing works
Self-citationLeveraging LLMs for Legacy Code Modernization: Evaluation of LLM-Generated Documentation
Diggs, C., Doyle, M., Madan, A., Scott, E. O., Escamilla, E., Zimmer, J., Nekoo, N., Ursino, P., Bartholf, M., Robin, Z., et al. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00027 · Cited by 9 · see citing works
Supporting Software Maintenance with Dynamically Generated Document Hierarchies
Dearstyne, K. R., Rodriguez, A. D., & Cleland‐Huang, J. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00046 · Cited by 8 · see citing works
Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
Kruse, H.-A., Puhlfürß, T., & Maalej, W. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00058 · Cited by 8 · see citing works
Sklavenitis, D., & Kalles, D. (2025) · Applied Sciences · doi:10.3390/app15137165 · Cited by 8 · see citing works
Delgado, D., Burgueño, L., & Clarisó, R. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112871 · Cited by 6 · see citing works
Test Code Generation for Telecom Software Systems Using Two-Stage Generative Model
Nabeel, M., Nimara, D. D., & Zanouda, T. (2024) · IEEE International Conference on Communications workshops · doi:10.1109/iccworkshops59551.2024.10615269 · Cited by 6 · see citing works
Learning a holistic and comprehensive code representation for code summarization
Yang, K., Wang, J., & Song, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111746 · Cited by 6 · see citing works
Al-Driven Automated Software Documentation Generation for Enhanced Development Productivity
Thota, S. R., Arora, S., & Gupta, S. (2024) · International Conference on Data Science and Network Security (ICDSNS) · doi:10.1109/icdsns62112.2024.10691221 · Cited by 6 · see citing works
Bappon, S. D., Mondal, S., & Roy, B. (2024) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam63643.2024.00013 · Cited by 5 · see citing works
QTC4SO: Automatic Question Title Completion for Stack Overflow
Zhou, Y., Yang, S., Chen, X., Zhang, Z., & Pei, J. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00011 · Cited by 4 · see citing works
Kim, J., Haddadian, G., & Kim, M. K. (2023) · Proceedings. · doi:10.22318/icls2023.633243 · Cited by 2 · see citing works
Epic-Level Text Generation with LLM through Auto-prompted Reinforcement Learning
Qi, Q., Ni, L., Wang, Z., Zhang, L., Liu, J., & Witbrock, M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650358 · Cited by 2 · see citing works
CLCoSum: Curriculum Learning-Based Code Summarization for Code Language Models
He, H., Wang, J., Cao, L., & Cai, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00035 · Cited by 1 · see citing works
Non-Linear Software Documentation with Interactive Code Examples
Nassif, M., & Robillard, M. P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702976 · Cited by 1 · see citing works
Prompt Learning for Source Code Summarization
Xu, T., Fang, C., Qian, H., Feng, X., & Sun, W. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00028 · Cited by 2 · see citing works
ZRB: A Novel Framework for Code Generation and Task Automation
Gunawan, G. F., & Amien, M. (2024) · International Conference on Smart Computing and Communications (ICSCC) · doi:10.1109/icscc62041.2024.10690452
Hasan, M. R., Alhanahnah, M., Stevens, C., & Bagheri, H. (2025) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn64029.2025.00023
Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings
Tsvetkov, P., Eliseeva, A., Dig, D., Bezzubov, A., Golubev, Y., Bryksin, T., & Zharov, Y. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00058
Automated Versioning for Software Releases: A Retrospective Study and a New Lightweight Approach
CHENG, X., CHONG, X., Weiyu, L. A. N., & Caimiao, Z. H. A. O. (2025) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2025mpp0003
Mixture-of-Experts Low-Rank Adaptation for Multilingual Code Summarization
Yu, T., Yuan, L., Huang, H., Wang, J., & Cai, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00186
Utilizing Chain of Thought to Generate Code from Challenging Programming Requirements
EmanAlofi, RehabIbrahim, & AmnehAlAbdi (2026) · International Conference on Consumer Electronics · doi:10.1109/icce67443.2026.11449835
Pominova, K., & Hao, H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27834-0_48
- Authors
- Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., & Zimmermann, T.
- Affiliations
- Zhejiang University (CN)
- Monash University (AU)
- Singapore Management University (SG)
- Microsoft (United States) (US)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2022-07-28
- Volume
- 31
- Issue
- 4
- Pages
- 1-28
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4281400706
- Cited by
- 33
- References
- 45
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 12 · 2023: 7 · 2022: 1
- FWCI
- 7.43
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- Funders
- National Science Foundation of China (U20A20173)
- National Research Foundation, Singapore
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- automatic evaluation metrics, human evaluation, code comment generation, commit message generation, meteors, BLEU, ROUGE-L, cider, spices, Java methods, annotation agreement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
