The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Abstract
During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer's time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers' program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.'s approach to assign developers' activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend ~58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers' experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,032
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Software engineering: A practitioner's approach
(1983) · Advances in Engineering Software (1978) · doi:10.1016/0141-1195(83)90118-3 · Cited by 8,234
Refactoring: Improving the Design of Existing Code
Fowler, M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45672-4_31 · Cited by 5,591
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Refactoring improving the design of existing code
Saca, M. A. (2017) · 2017 IEEE 37th Central America and Panama Convention (CONCAPAN XXXVII) · doi:10.1109/concapan.2017.8278488 · Cited by 2,054
The psychology of computer programming
(1999) · Choice Reviews Online · doi:10.5860/choice.36-4535 · Cited by 962
Research works printed but not linked: 3
- whitaker, 1999, What causes IT workers to leave, Manag Rev, 88, 8
- spencer, 2009, Card Sorting Designing Usable Categories
- fjeldstad, 1983, Application program maintenance study: Report to our respondents, Proc GUIDE
Cited by
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2018) · Proceedings of the 26th Conference on Program Comprehension · doi:10.1145/3196321.3196334 · Cited by 621
Self-citationFew-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240
API method recommendation without worrying about the task-API knowledge gap
Huang, Q., Xia, X., Xing, Z., Lo, D., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238191 · Cited by 192
Self-citationHow do developers utilize source code from stack overflow?
Wu, Y., Wang, S., Bezemer, C., & Inoue, K. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9634-5 · Cited by 129
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 148
Metadata
- Authors
- Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Singapore Management University (SG)
- Australian National University (AU)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2018-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 951-976
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2741561716
- Cited by
- 326
- References
- 64
- Citations by year
- 2026: 30 · 2025: 37 · 2024: 54 · 2023: 62 · 2022: 60 · 2021: 38 · 2020: 22 · 2019: 14 · 2018: 8
- FWCI
- 22.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61602403, 61572426)
- National Key Technology R&D Program
- Ministry of Science and Technology of China (2015BAH17F01)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- program comprehension, professional developers, developer experience, project phases, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring Program Comprehension: A Large-Scale Field Study with Professionals. https://app.cochat.ai/papers/10.1109/tse.2017.2734091
- The paper
- Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018). Measuring Program Comprehension: A Large-Scale Field Study with Professionals. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2017.2734091
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.