Measuring Program Comprehension: A Large-Scale Field Study with Professionals
The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer's time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers' program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.'s approach to assign developers' activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend ~58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers' experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,031
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Software engineering: A practitioner's approach
(1983) · Advances in Engineering Software (1978) · doi:10.1016/0141-1195(83)90118-3 · Cited by 8,234
Refactoring: Improving the Design of Existing Code
Fowler, M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45672-4_31 · Cited by 5,591
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Refactoring improving the design of existing code
Saca, M. A. (2017) · 2017 IEEE 37th Central America and Panama Convention (CONCAPAN XXXVII) · doi:10.1109/concapan.2017.8278488 · Cited by 2,054
The psychology of computer programming
(1999) · Choice Reviews Online · doi:10.5860/choice.36-4535 · Cited by 962
Research works printed but not linked: 3
- whitaker, 1999, What causes IT workers to leave, Manag Rev, 88, 8
- spencer, 2009, Card Sorting Designing Usable Categories
- fjeldstad, 1983, Application program maintenance study: Report to our respondents, Proc GUIDE
Cited by
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2018) · Proceedings of the 26th Conference on Program Comprehension · doi:10.1145/3196321.3196334 · Cited by 621
Self-citationFew-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240
API method recommendation without worrying about the task-API knowledge gap
Huang, Q., Xia, X., Xing, Z., Lo, D., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238191 · Cited by 192
Self-citationHow do developers utilize source code from stack overflow?
Wu, Y., Wang, S., Bezemer, C.-P., & Inoue, K. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9634-5 · Cited by 127
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 148
Today was a good day: The daily life of software developers
Meyer, A. N., Barr, E. T., Bird, C., & Zimmermann, T. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2904957 · Cited by 123
Bao, L., Li, T., Xia, X., Zhu, K., Li, H., & Yang, X. (2022) · Science China Information Sciences · doi:10.1007/s11432-020-3278-4 · Cited by 207
Self-citationCode Structure–Guided Transformer for Source Code Summarization
Gao, S., Gao, C., He, Y., Zeng, J., Nie, L., Xia, X., & Lyu, M. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3522674 · Cited by 113
Self-citationA Large-Scale Empirical Study on Code-Comment Inconsistencies
Wen, F., Nagy, C., Bavota, G., & Lanza, M. (2019) · 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00019 · Cited by 105
Automatic Code Documentation Generation Using GPT-3
Khan, J. Y., & Uddin, G. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559548 · Cited by 96
A Human Study of Comprehension and Code Summarization
Stapleton, S., Gambhir, Y., LeClair, A., Eberhart, Z., Weimer, W., Leach, K., & Huang, Y. (2020) · Proceedings of the 28th International Conference on Program Comprehension · doi:10.1145/3387904.3389258 · Cited by 77
An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability
Barón, M. M., Wyrich, M., & Wagner, S. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3410636 · Cited by 121
Retrieve and refine: exemplar-based neural comment generation
Wei, B., Li, Y., Li, G., Xia, X., & Jin, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416578 · Cited by 69
Self-citationImproving code readability classification using convolutional neural networks
Mi, Q., Keung, J., Xiao, Y., Mensah, S., & Gao, Y. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.07.006 · Cited by 76
EditSum: A Retrieve-and-Edit Framework for Source Code Summarization
Li, J. A., Li, Y., Li, G., Hu, X., Xia, X., & Jin, Z. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678724 · Cited by 62
Self-citationA Survey of Automatic Source Code Summarization
Zhang, C., Wang, J., Zhou, Q., Xu, T. X., Tang, K., Gui, H., & Liu, F. (2022) · Symmetry · doi:10.3390/sym14030471 · Cited by 103
A decade of code comment quality assessment: A systematic literature review
Rani, P., Blasi, A., Stulova, N., Panichella, S., Gorla, A., & Nierstrasz, O. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111515 · Cited by 80
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Why My Code Summarization Model Does Not Work: Code Comment Improvement with Category Prediction
Chen, Q., Xia, X., Hu, H., Lo, D., & Li, S. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3434280 · Cited by 86
Self-citationAUGER: automatically generating review comments with pre-training models
Li, L., Yang, L., Jiang, H., Yan, J., Luo, T., Hua, Z., Liang, G., & Zuo, C. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549099 · Cited by 57
Evaluating Code Readability and Legibility: An Examination of Human-centric Studies
Oliveira, D., Bruno, R., Madeiral, F., & Castor, F. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00041 · Cited by 89
Retrieve and Refine: Exemplar-Based Neural Comment Generation
Wei, B. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00152 · Cited by 25
A Fast Clustering Algorithm for Modularization of Large-Scale Software Systems
Teymourian, N., Izadkhah, H., & Isazadeh, A. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3022212 · Cited by 60
ActionNet: Vision-Based Workflow Action Recognition From Programming Screencasts
Zhao, D., Xing, Z., Chen, C., Xia, X., & Li, G. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00049 · Cited by 58
Self-citationProject-Level Encoding for Neural Source Code Summarization of Subroutines
Bansal, A., Haque, S., & McMillan, C. (2021) · 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00032 · Cited by 46
40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study
Wyrich, M., Bogner, J., & Wagner, S. (2023) · ACM Computing Surveys · doi:10.1145/3626522 · Cited by 75
Enabling Good Work Habits in Software Developers through Reflective Goal-Setting
Meyer, A. N., Murphy, G. C., Zimmermann, T., & Fritz, T. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2938525 · Cited by 43
On the Significance of Category Prediction for Code-Comment Synchronization
Yang, Z., Keung, J., Yu, X., Xiao, Y., Jin, Z. G., & Zhang, J. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3534117 · Cited by 48
M2TS: multi-scale multi-modal approach based on transformer for source code summarization
Gao, Y., & Lyu, C. (2022) · Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3524610.3527907 · Cited by 43
From Code Complexity Metrics to Program Comprehension
Feitelson, D. G. (2023) · Communications of the ACM · doi:10.1145/3546576 · Cited by 60
An Extractive-and-Abstractive Framework for Source Code Summarization
Sun, W., Fang, C., Chen, Y., Zhang, Q., Tao, G., You, Y., Han, T., Ge, Y., Hu, Y., Luo, B., et al. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3632742 · Cited by 69
SeCNN: A semantic CNN parser for code comment generation
Li, Z., Wu, Y., Peng, B., Chen, X., Sun, Z., Liu, Y., & Yu, D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111036 · Cited by 49
Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
Lo, D. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00010 · Cited by 40
Self-citationIvie: Lightweight Anchored Explanations of Just-Generated Code
Yan, L., Hwang, A., Wu, Z., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642239 · Cited by 66
Yang, G., Chen, X., Cao, J., Xu, S., Cui, Z., Yu, C., & Liu, K. (2021) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa52907.2021.00013 · Cited by 45
Source Code Summarization with Structural Relative Position Guided Transformer
Gong, Z., Gao, C., Wang, Y., Gu, W., Peng, Y., & Xu, Z. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00013 · Cited by 54
BashExplainer: Retrieval-Augmented Bash Code Comment Generation based on Fine-tuned CodeBERT
Yu, C., Yang, G., Chen, X., Liu, K., & Zhou, Y. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00016 · Cited by 54
Tornhill, A., & Borg, M. (2022) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3524843.3528091 · Cited by 78
Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization
Mastropaolo, A., Ciniselli, M., Penta, M. D., & Bavota, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639174 · Cited by 76
GypSum: learning hybrid representations for code summarization
Wang, Y., Dong, Y., Lu, X., & Zhou, A. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527903 · Cited by 44
Razzaq, A., Buckley, J., Lai, Q., Yu, T., & Botterweck, G. (2025) · ACM Computing Surveys · doi:10.1145/3687299 · Cited by 36
DeepCommenter: a deep code comment generation tool with hybrid lexical and syntactical information
Li, B., Yan, M., Xia, X., Hu, X., Li, G., & Lo, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417926 · Cited by 35
Self-citationDualSC: Automatic Generation and Summarization of Shellcode via Transformer and Dual Learning
Yang, G., Chen, X., Zhou, Y., & Yu, C. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00052 · Cited by 44
Correlating Automated and Human Evaluation of Code Documentation Generation Quality
Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., & Zimmermann, T. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3502853 · Cited by 33
Self-citationDealing with software complexity in individual‐based models
Vedder, D., Ankenbrand, M. J., & Cabral, J. S. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13716 · Cited by 37
Augmenting Java method comments generation with context information based on neural networks
Zhou, Y., Yan, X. C., Yang, W., Chen, T., & Huang, Z. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.07.087 · Cited by 34
API2Com: On the Improvement of Automatically Generated Code Comments Using API Documentations
Shahbazi, R., Sharma, R., & Fard, F. H. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00049 · Cited by 46
TreeCaps: Tree-Based Capsule Networks for Source Code Processing
Bui, N. D. Q., Yu, Y., & Jiang, L. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16074 · Cited by 52
Detecting Code Comment Inconsistency using Siamese Recurrent Network
Rabbi, F., & Siddik, M. S. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389286 · Cited by 30
A Review on Source Code Documentation
Rai, S., Belwal, R. C., & Gupta, A. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3519312 · Cited by 27
Automatic Code Summarization: A Systematic Literature Review
Zhu, Y., & Pan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04352 · Cited by 76
Understanding How Programmers Can Use Annotations on Documentation
Horvath, A., Liu, M. X., Hendriksen, R., Shannon, C., Paterson, E., Jawad, K., Macvean, A., & Myers, B. A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502095 · Cited by 26
Wyrich, M., Preikschat, A., Graziotin, D., & Wagner, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00055 · Cited by 24
Kuang, L., Zhou, C., & Yang, X. (2022) · Automated Software Engineering · doi:10.1007/s10515-022-00341-1 · Cited by 28
An Accurate Identifier Renaming Prediction and Suggestion Approach
Zhang, J., Luo, J., Liang, J., Gong, L., & Huang, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3603109 · Cited by 24
Esale: Enhancing Code-Summary Alignment Learning for Source Code Summarization
Fang, C., Sun, W., Chen, Y., Chen, X., Wei, Z., Zhang, Q., You, Y., Luo, B., Liu, Y., & Chen, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3422274 · Cited by 39
How SonarQube-identified technical debt is prioritized: An exploratory case study
Alfayez, R., Winn, R., Alwehaibi, W., Venson, E., & BOEHM, B. W. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107147 · Cited by 35
Generation-based Code Review Automation: How Far Are We?
Zhou, X., Kim, K., Xu, B., Han, D., He, J., & Lo, D. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00036 · Cited by 57
Self-citationMeasuring program comprehension
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182538 · Cited by 22
Self-citationTowards using visual, semantic and structural features to improve code readability classification
Mi, Q., Hao, Y., Ou, L., & Ma, W. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111454 · Cited by 34
Long, T., Xie, Y., Chen, X., Zhang, W., Cao, Q., & Yu, Y. H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20522 · Cited by 25
Interpretation-based Code Summarization
Geng, M., Wang, S., Dong, D., Wang, H., Cao, S., Zhang, K., & Jin, Z. G. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00026 · Cited by 29
Deep Code-Comment Understanding and Assessment
Wang, D., Guo, Y., Dong, W., Wang, Z., Liu, H., & Li, S. (2019) · IEEE Access · doi:10.1109/access.2019.2957424 · Cited by 22
Self-citationSupporting Code Comprehension via Annotations: Right Information at the Right Time and Place
Adeli, M., NELSON, N. M., Chattopadhyay, S., Coffey, H., Henley, A. Z., & Sarma, A. (2020) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127264 · Cited by 26
Endres, M., Fansher, M., Shah, P., & Weimer, W. R. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468583 · Cited by 27
Lavazza, L., Abualkishik, A. Z., Liu, G., & Morasca, S. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111561 · Cited by 38
Personality and Project Success: Insights from a Large-Scale Study with Professionals
Xia, X., Lo, D., Bao, L., Sharma, A. B., & Li, S. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.50 · Cited by 26
Self-citationA review of automatic source code summarization
Zhang, X., Hou, X., Qiao, X., & Song, W. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10553-6 · Cited by 34
Relating Reading, Visualization, and Coding for New Programmers: A Neuroimaging Study
Endres, M., Karas, Z., Hu, X., Kovelman, I., & Weimer, W. R. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00062 · Cited by 26
Cheng, W., Hu, P., Wei, S., & Mo, R. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106987 · Cited by 18
An empirical study on software understandability and its dependence on code characteristics
Lavazza, L., Morasca, S., & Gatto, M. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10396-7 · Cited by 35
Collaborative program comprehension via software visualization in extended reality
Krause-Glau, A., Hansen, M., & Hasselbring, W. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107007 · Cited by 17
Understanding Large-Scale Software – A Hierarchical View
Levy, O., & Feitelson, D. G. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00047 · Cited by 26
Summarizing source code with hierarchical code representation
Zhou, Z., Yu, H., Fan, G., Huang, Z., & Yang, X. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106761 · Cited by 16
Fine-grained code-comment semantic interaction analysis
Geng, M., Wang, S., Dong, D., Gu, S., Peng, F., Ruan, W., & Liao, X. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527887 · Cited by 20
A graph-based code representation method to improve code readability classification
Mi, Q., Zhan, Y., Weng, H., Bao, Q., Cui, L., & Ma, W. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10319-6 · Cited by 26
Collaborative Software Visualization for Program Comprehension
Krause-Glau, A., Bader, M., & Hasselbring, W. (2022) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft55257.2022.00016 · Cited by 15
Borg, M., Tornhill, A., & Mones, E. (2023) · Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3593434.3593480 · Cited by 18
Ding, X., Peng, R., Chen, X., Huang, Y., Bian, J., & Zheng, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652156 · Cited by 26
On the Effectiveness of Large Language Models in Statement-level Code Summarization
Zhu, J., Miao, Y., Xu, T., Zhu, J., & Sun, X. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00030 · Cited by 20
A Model to Detect Readability Improvements in Incremental Changes
Roy, D., Fakhoury, S., Lee, J., & Arnaoudova, V. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389255 · Cited by 17
Understanding large-scale software systems – structure and flows
Levy, O., & Feitelson, D. G. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09938-8 · Cited by 22
Generating summaries for methods of event-driven programs: An Android case study
Aghamohammadi, A., Izadi, M., & Heydarnoori, A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110800 · Cited by 16
Towards Retrieval-Based Neural Code Summarization: A Meta-Learning Approach
Zhou, Z., Yu, H., Fan, G., Huang, Z., & Yang, K. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3238161 · Cited by 21
Unraveling the Impact of Code Smell Agglomerations on Code Stability
Santana, A., Figueiredo, E., & Pereira, J. A. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00049 · Cited by 20
Understanding Code Understandability Improvements in Code Reviews
Oliveira, D., Santos, R., de Oliveira, B., Monperrus, M., Castor, F., & Madeiral, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3453783 · Cited by 20
Defect prediction using deep learning with Network Portrait Divergence for software evolution
Walunj, V., Gharibi, G., Alanazi, R. K. M., & Lee, Y. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10147-0 · Cited by 21
Pseudocode Generation from Source Code Using the BART Model
Alokla, A., Gad, W. K., Nazih, W., Aref, M., & Salem, A.-B. M. (2022) · Mathematics · doi:10.3390/math10213967 · Cited by 20
Qiao, Y., Shihab, M. I. H., & Hundhausen, C. D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3785366 · Cited by 31
HatCUP: hybrid analysis and attention based just-in-time comment updating
Zhu, H., He, X., & Xu, L. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527901 · Cited by 16
Understanding and Supporting Debugging Workflows in Multiverse Analysis
Gu, K., Jun, E., & Althoff, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581099 · Cited by 19
An exploratory study on the identification and evaluation of bad smell agglomerations
Santana, A., Cruz, D., & Figueiredo, E. (2021) · ACM Symposium on Applied Computing · doi:10.1145/3412841.3442003 · Cited by 11
The Impact of AI on Cybersecurity Defense Mechanisms: Future Trends and ChallengesPreprint
Gilbert, C., & Gilbert, M. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5258783 · Cited by 17
van der Werf, V., Swidan, A., Hermans, F., Specht, M. M., & Aivaloglou, E. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640069 · Cited by 11
Learning to Generate Structured Code Summaries From Hybrid Code Context
Zhou, Z., Li, M., Yu, H., Fan, G., Yang, P., & Huang, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3439562 · Cited by 13
MMF3: Neural Code Summarization Based on Multi-Modal Fine-Grained Feature Fusion
Ma, Z., Gao, Y., Lyu, L., & Lyu, C. (2022) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3544902.3546251 · Cited by 12
Leveraging Large Language Models to Boost Dafny’s Developers Productivity
Silva, Á. F., Mendes, A., & Ferreira, J. F. (2024) · IEEE/ACM International Conference on Formal Methods in Software Engineering (FormaliSE) · doi:10.1145/3644033.3644374 · Cited by 27
An exploratory evaluation of code smell agglomerations
Santana, A., Figueiredo, E., Pereira, J. A., & Garcia, A. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09680-6 · Cited by 11
ClassSum: a deep learning model for class-level code summarization
Li, M., Yu, H., Fan, G., Zhou, Z., & Huang, J. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07877-z · Cited by 9
Enhancing code summarization with action word prediction
Li, M., Yu, H., Fan, G., Zhou, Z., & Huang, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126777 · Cited by 9
Metadata
- Authors
- Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Singapore Management University (SG)
- Australian National University (AU)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2018-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 951-976
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2741561716
- Cited by
- 326
- References
- 64
- Citations by year
- 2026: 30 · 2025: 37 · 2024: 54 · 2023: 62 · 2022: 60 · 2021: 38 · 2020: 22 · 2019: 14 · 2018: 8
- FWCI
- 22.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61602403, 61572426)
- National Key Technology R&D Program
- Ministry of Science and Technology of China (2015BAH17F01)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- program comprehension, professional developers, developer experience, project phases, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.