A Human Study of Comprehension and Code Summarization
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Software developers spend a great deal of time reading and understanding code that is poorly-documented, written by other developers, or developed using differing styles. During the past decade, researchers have investigated techniques for automatically documenting code to improve comprehensibility. In particular, recent advances in deep learning have led to sophisticated summary generation techniques that convert functions or methods to simple English strings that succinctly describe that code's behavior. However, automatic summarization techniques are assessed using internal metrics such as BLEU scores, which measure natural language properties in translational models, or ROUGE scores, which measure overlap with human-written text. Unfortunately, these metrics do not necessarily capture how machine-generated code summaries actually affect human comprehension or developer productivity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards a theory of the comprehension of computer programs
Brooks, R. (1983) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(83)80031-5 · Cited by 756
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514 · cited 2×
“For this reason, many proposed code summarization techniques are accompanied by a human study, in which programmers are asked to judge the quality of generated comments [23, 46, 62].”
BackgroundA Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386 · cited 2×
“Second, BLEU was designed specifically to evaluate machine translation systems [53].”
BackgroundMethodMeasuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326
Supporting program comprehension with source code summarization
Haiduc, S., Aponte, J., & Marcus, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2 · doi:10.1145/1810295.1810335 · Cited by 291 · cited 1×
“When baseline summaries are not readily available, researchers may first perform a human study in which programmers are asked to generate summaries to serve as baselines [22, 54].”
BackgroundQuestions programmers ask during software evolution tasks
Sillito, J., Murphy, G. C., & De Volder, K. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181779 · Cited by 270 · cited 3×
“Sillito et al. [61] performed empirical studies to determine the most common types of questions programmers ask during software evolution tasks.”
BackgroundMethodInfluentialAutomatically documenting program changes
Buse, R. P. L., & Weimer, W. R. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859005 · Cited by 219
Self-citationImproving automated source code summarization via an eye-tracking study of programmers
Rodeghero, P., McMillan, C., McBurney, P. W., Bosch, N., & D'Mello, S. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568247 · Cited by 198 · cited 2×
“When baseline summaries are not readily available, researchers may first perform a human study in which programmers are asked to generate summaries to serve as baselines [22, 54].”
BackgroundMethod(2017) · doi:10.1145/3106237.3106268 · Cited by 117
Cited by
Semantic similarity metrics for evaluating source code summarization
Haque, S., Eberhart, Z., Bansal, A., & McMillan, C. (2022) · Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3524610.3527909 · Cited by 116
“The dominant metric is currently BLEU, and weaknesses of BLEU have been high-lighted for several years [11], though it is not clear if any existing metric should replace it.”
BackgroundResultInfluentialSelf-citationReassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106
“[52] explore how automatically generated summaries effect code comprehension and show that participants per-”
Background(2023) · doi:10.1145/3510003.3510060 · Cited by 88
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Correlating Automated and Human Evaluation of Code Documentation Generation Quality
Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., & Zimmermann, T. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3502853 · Cited by 33
Metadata
- Authors
- Stapleton, S., Gambhir, Y., LeClair, A., Eberhart, Z., Weimer, W., Leach, K., & Huang, Y.
- Venue
- Proceedings of the 28th International Conference on Program Comprehension
- Publisher
- ACM
- Published
- 2020-07-13
- Pages
- 2-13
- OpenAlex
- W3085939759
- Cited by
- 77
- References
- 9
- FWCI
- 7.74
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Human Study of Comprehension and Code Summarization. https://app.cochat.ai/papers/10.1145/3387904.3389258
- The paper
- Stapleton, S., Gambhir, Y., LeClair, A., Eberhart, Z., Weimer, W., Leach, K., & Huang, Y. (2020). A Human Study of Comprehension and Code Summarization. Proceedings of the 28th International Conference on Program Comprehension. https://doi.org/10.1145/3387904.3389258
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.