Measuring Program Comprehension: A Large-Scale Field Study with Professionals
The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer's time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers' program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.'s approach to assign developers' activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend ~58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers' experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,042
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Software engineering: A practitioner's approach
(1983) · Advances in Engineering Software (1978) · doi:10.1016/0141-1195(83)90118-3 · Cited by 8,234
Refactoring: Improving the Design of Existing Code
Fowler, M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45672-4_31 · Cited by 5,591
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Refactoring improving the design of existing code
Saca, M. A. (2017) · 2017 IEEE 37th Central America and Panama Convention (CONCAPAN XXXVII) · doi:10.1109/concapan.2017.8278488 · Cited by 2,054
The psychology of computer programming
(1999) · Choice Reviews Online · doi:10.5860/choice.36-4535 · Cited by 962
Research works printed but not linked: 3
- whitaker, 1999, What causes IT workers to leave, Manag Rev, 88, 8
- spencer, 2009, Card Sorting Designing Usable Categories
- fjeldstad, 1983, Application program maintenance study: Report to our respondents, Proc GUIDE
Cited by
Items 301–325 of 325
Key-augmented neural triggers for knowledge sharing
Wolf, A., Palma, M. E., Rani, P., & Gall, H. C. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112850 · Cited by 1
Tulkunova, N., Jolha, F., Egorova, A., Bugaev, G., Maksimova, A., Okoneshnikov, D., & Zuev, E. (2025) · doi:10.1145/3788149.3788237
Zhong, M., Huizhu, L., Ting, Y., Kang, C., Xi, C., Yi, J., & Cheng, Y. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03045-0
Code Together, Apart: Teaching Asynchronous Team Development Workflows with GitHub
Dana, K., Sen, A., & Joy, J. M. (2026) · Information Systems Education Journal · doi:10.62273/zrxr5679
Towards Transferring Text Comprehension Techniques to Program Comprehension
Dreyer, T., Michel, J., & Bockisch, C. M. (2026) · International Conference on Art, Science, and Engineering of Programming (<Programming>) · doi:10.1145/3801119.3801126
Optimizing Knowledge Utilization for Multi-Intent Comment Generation With Large Language Models
Li, S., Wang, Z., Du, X., Wu, Z., Yu, J., & Chen, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3687241
Self-citationAdaFuse: An Adaptive Retrieval-and-Generation Framework for Code Summarization
国贺梅, Su, H., Gao, H., & Xing, Y. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-4897-2_75
Structure-Aware Transformer for Code Snippet Summarization
Xing, Y., Su, H., Gao, H., & 国贺梅 (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-4897-2_19
Kumar, J., Tanguturi, M. R., Janapati, L. S., & Chimalakonda, S. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113010 · Cited by 1
Chapagain, J., & Rus, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29744-0_3
How Robustly Do LLMs Understand Execution Semantics?
Spieß, C. K., Dévanbu, P., & Barr, E. T. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814919 · Cited by 3
Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework
Aguiar, L., Freitas, M., Paixão, M. H. E., & do Carmo, R. A. F. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788728
Automatic Validation of LLM-Generated Code with Prompt Paraphrasing
Wang, X., & Zhu, D. K. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786809 · Cited by 1
Leveraging Design-Aware Context in Large Language Models for Code Comment Generation
Mitra, A., Majumdar, S., Mukhopadhyay, A., Das, P. P., Clough, P. D., & Chakrabarti, P. P. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786825
Adaptive AI Productivity Assistant for Developers in Visual Studio Code
Ranasinghe, R. A. N. D., Veenath, G. D. T., Silva, K. C. R., Beligolla, B. G. D. N., Wijendra, D., & Krishara, J. (2026) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn70047.2026.11594203
CrossLink: A Tool for Cross-Language (Java-C/C++) Code Comprehension
Sawarkar, A., Arrepu, A., & Chimalakonda, S. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794830
Learning Developer‐Code Contribution Relationship to Improve Code Readability Assessment
Mi, Q., Wang, Y., & Li, J. (2026) · Journal of Software Evolution and Process · doi:10.1002/smr.70168
Identifying key classes using biterm-sensitive PageRank with motif-based coreness
Pan, W., Cui, X., Ming, H., Yang, Z., Chen, J., Jiang, B., & Almukhtar, M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3844140 · Cited by 1
Comprehensibility-guided Automated Code Refactoring
Silva, N. D. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787672
Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot
Liu, F., & McKinnon, T. (2026) · Communications of the ACM · doi:10.1145/3797488 · Cited by 1
Alqahtani, S. O., & Ghoneim, A. (2026) · Applied Sciences · doi:10.3390/app16199845
ForkOps: Operating Fleets of Generated Software Variants
Bindschaedler, L. (2026) · doi:10.1145/3844178.3844641 · Cited by 1
Traceable RAG for Legacy COBOL Comprehension: Integrating Architectural and Semantic Analysis
Kader, H. A., Lilaj, E., Calamo, M., Coluzzi, A., Monti, F., Palumbo, G., Carotenuto, G., & Mecella, M. (2026) · doi:10.1145/3843775.3844545
KnowJIT: Knowledge-Augmented Just-in-Time Defect Prediction and Repair via In-Context Learning
Wu, D., Li, W., Lv, G., Zhao, X., Yang, Y., & Dong, Z. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3744216
Feeling Easier, Working Harder: Cognitive Load in AI-Assisted Code Comprehension
White, M., Rose, M., Buckman, M., McDonnell, A., Ortega, F., & Trinkenreich, B. (2026) · arXiv · doi:10.48550/arxiv.2611.12019
Metadata
- Authors
- Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Singapore Management University (SG)
- Australian National University (AU)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2018-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 951-976
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2741561716
- Cited by
- 326
- References
- 64
- Citations by year
- 2026: 30 · 2025: 37 · 2024: 54 · 2023: 62 · 2022: 60 · 2021: 38 · 2020: 22 · 2019: 14 · 2018: 8
- FWCI
- 22.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61602403, 61572426)
- National Key Technology R&D Program
- Ministry of Science and Technology of China (2015BAH17F01)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- program comprehension, professional developers, developer experience, project phases, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.