Measuring Program Comprehension: A Large-Scale Field Study with Professionals
The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
During software development and maintenance, developers spend a considerable amount of time on program comprehension activities. Previous studies show that program comprehension takes up as much as half of a developer's time. However, most of these studies are performed in a controlled setting, or with a small number of participants, and investigate the program comprehension activities only within the IDEs. However, developers' program comprehension activities go well beyond their IDE interactions. In this paper, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We follow Minelli et al.'s approach to assign developers' activities into four categories: navigation, editing, comprehension, and other. We then measure the comprehension time by calculating the time that developers spend on program comprehension, e.g., inspecting console and breakpoints in IDE, or reading and understanding tutorials in web browsers. Using this approach, we can perform a more realistic investigation of program comprehension activities, through a field study of program comprehension in practice across a total of seven real projects, on 78 professional developers, and amounting to 3,148 working hours. Our study leverages interaction data that is collected across many applications by the developers. Our study finds that on average developers spend ~58 percent of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers' experience, and project phase on the time that is spent on program comprehension, and we find senior developers spend significantly less percentages of time on program comprehension than junior developers. Our study also highlights the importance of several research directions needed to reduce program comprehension time, e.g., building automatic detection and improvement of low quality code and documentation, construction of software-engineering-specific search engines, designing better IDEs that help developers navigate code and browse information more efficiently, etc.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,031
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Software engineering: A practitioner's approach
(1983) · Advances in Engineering Software (1978) · doi:10.1016/0141-1195(83)90118-3 · Cited by 8,234
Refactoring: Improving the Design of Existing Code
Fowler, M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45672-4_31 · Cited by 5,591
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Refactoring improving the design of existing code
Saca, M. A. (2017) · 2017 IEEE 37th Central America and Panama Convention (CONCAPAN XXXVII) · doi:10.1109/concapan.2017.8278488 · Cited by 2,054
The psychology of computer programming
(1999) · Choice Reviews Online · doi:10.5860/choice.36-4535 · Cited by 962
Significance Testing of the Spearman Rank Correlation Coefficient
Zar, J. H. (1972) · Journal of the American Statistical Association · doi:10.1080/01621459.1972.10481251 · Cited by 689
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685
Object-oriented design heuristics
Riel, A. J. (1996) · Cited by 660
Maintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Program comprehension during software maintenance and evolution
Von Mayrhauser, A., & Vans, A. M. (1995) · Computer · doi:10.1109/2.402076 · Cited by 575
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514
Requirements engineering and agile software development
Paetsch, F., Eberlein, A., & Maurer, F. (2004) · WET ICE 2003. Proceedings. Twelfth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2003. · doi:10.1109/enabl.2003.1231428 · Cited by 459
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452
Significance Testing of the Spearman Rank Correlation Coefficient
Zar, J. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284441 · Cited by 396
Program understanding: Challenge for the 1990s
Corbi, T. A. (1989) · IBM Systems Journal · doi:10.1147/sj.282.0294 · Cited by 388
Polymetric views - A lightweight visual approach to reverse engineering
Lanza, M., & Ducasse, S. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1232284 · Cited by 373
Designing the whyline: a debugging interface for asking questions about program behavior
Ko, A. J., & Myers, B. A. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985712 · Cited by 361
Automatic generation of natural language summaries for Java classes
Moreno, L., Aponte, J., Sridhara, G., Marcus, A., Pollock, L., & Vijay-Shanker, K. (2013) · 2013 21st International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2013.6613830 · Cited by 331
Whats in a Name? A Study of Identifiers
Lawrie, D., Morrell, C., Feild, H., & Binkley, D. (2006) · 14th IEEE International Conference on Program Comprehension (ICPC'06) · doi:10.1109/icpc.2006.51 · Cited by 248
AutoComment: Mining question and answer sites for automatic comment generation
Wong, E., Yang, J., & Tan, L. (2013) · 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693113 · Cited by 245
Jiang, J. J., & Klein, G. (1999) · Journal of Management Information Systems · doi:10.1080/07421222.1999.11518262 · Cited by 244
Sourcerer: mining and searching internet-scale software repositories
Linstead, E., Bajracharya, S., Ngo, T., Rigor, P., Lopes, C., & Baldi, P. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0118-x · Cited by 237
Automatic documentation generation via source code summarization of method context
McBurney, P. W., & McMillan, C. (2014) · Proceedings of the 22nd International Conference on Program Comprehension · doi:10.1145/2597008.2597149 · Cited by 220
Sourcerer: a search engine for open source code supporting structure-based search
Bajracharya, S., Ngo, T., Linstead, E., Dou, Y., Rigor, P., Baldi, P., & Lopes, C. (2006) · Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications · doi:10.1145/1176617.1176671 · Cited by 219
Automatic query reformulations for text retrieval in software engineering
Haiduc, S., Bavota, G., Marcus, A., Oliveto, R., Lucia, A. D., & Menzies, T. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606630 · Cited by 199
Murphy-Hill, E., Zimmermann, T., & Nagappan, N. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568226 · Cited by 183
Automatically detecting and describing high level actions within methods
Sridhara, G., Pollock, L., & Vijay-Shanker, K. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985808 · Cited by 182
I Know What You Did Last Summer - An Investigation of How Developers Spend Their Time
Minelli, R., Mocci, A., & Lanza, M. (2015) · 2015 IEEE 23rd International Conference on Program Comprehension · doi:10.1109/icpc.2015.12 · Cited by 181
On the Comprehension of Program Comprehension
Maalej, W., Tiarks, R., Roehm, T., & Koschke, R. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2622669 · Cited by 180
How do professional developers comprehend software?
Roehm, T., Tiarks, R., Koschke, R., & Maalej, W. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227188 · Cited by 178
Code bubbles: rethinking the user interface paradigm of integrated development environments
Bragdon, A., Reiss, S. P., Zeleznik, R., Karumuri, S., Cheung, W., Kaplan, J., Coleman, C., Adeputra, F., & LaViola, J. J. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806866 · Cited by 173
Boslaugh, S., & Watters, P. (2008) · Cited by 172
Automatic Query Reformulations for Text Retrieval in Software Engineering
Haiduc, S., Bavota, G., Marcus, A., Oliveto, R., Lucia, A. D., & Menzies, T. (2013) · Archivio Istituzionale della Ricerca - Università degli Studi del Molise · Cited by 167
How do professional developers comprehend software
Roehm, T., Tiarks, R., Koschke, R., & Maalej, W. (2012) · doi:10.5555/2337223.2337254 · Cited by 167
An Eye Tracking Study on camelCase and under_score Identifier Styles
Sharif, B., & Maletic, J. I. (2010) · 2010 IEEE 18th International Conference on Program Comprehension · doi:10.1109/icpc.2010.41 · Cited by 144
Principles of software engineering and design
(1980) · Computer Languages · doi:10.1016/0096-0551(80)90070-3 · Cited by 142
Measuring and modeling programming experience
Siegmund, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2014) · Empirical Software Engineering · doi:10.1007/s10664-013-9286-4 · Cited by 141
How practitioners perceive the relevance of software engineering research
Lo, D., Nagappan, N., & Zimmermann, T. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786809 · Cited by 135
Self-citationQuery Expansion Based on Crowd Knowledge for Code Search
Nie, L., Jiang, H., Ren, Z., Sun, Z., & Li, X. (2016) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2016.2560165 · Cited by 125
Impact of developer turnover on quality in open-source software
Foucault, M., Palyart, M., Blanc, X., Murphy, G. C., & Falleri, J.-R. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786870 · Cited by 116
Binkley, D., Davis, M., Lawrie, D., & Morrell, C. (2009) · 2009 IEEE 17th International Conference on Program Comprehension · doi:10.1109/icpc.2009.5090039 · Cited by 115
Fundamentals of Software Engineering
O’Regan, G. (2019) · Undergraduate topics in computer science · doi:10.1007/978-3-030-28494-7_2 · Cited by 101
Enriching Documents with Examples: A Corpus Mining Approach
Kim, J.-H., Lee, S.-H., Hwang, S.-W., & Kim, S. (2013) · ACM Transactions on Information Systems · doi:10.1145/2414782.2414783 · Cited by 97
Seahawk: Stack Overflow in the IDE
Ponzanelli, L., Bacchelli, A., & Lanza, M. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606701 · Cited by 95
Using IR methods for labeling source code artifacts: Is it worthwhile?
De Lucia, A., Di Penta, M., Oliveto, R., Panichella, A., & Panichella, S. (2012) · 2012 20th IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240488 · Cited by 94
Lemos, O. A. L., Bajracharya, S. K., Ossher, J., Morla, R., Masiero, P. C., Baldi, P., & Lopes, C. V. (2007) · doi:10.1145/1321631.1321726 · Cited by 85
Seahawk: Stack Overflow in the IDE
Ponzanelli, L., Bacchelli, A., & Lanza, M. (2016) · Cited by 82
Rigby, P. C., Zhu, Y. C., Donadelli, S. M., & Mockus, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884851 · Cited by 76
Rahman, M. M., Yeasmin, S., & Roy, C. K. (2014) · 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE) · doi:10.1109/csmr-wcre.2014.6747170 · Cited by 75
On the relationship between comment update practices and Software Bugs
Ibrahim, W. M., Bettenburg, N., Adams, B., & Hassan, A. E. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2011.09.019 · Cited by 73
Self-citationSuccession: Measuring transfer of code and developer productivity
Mockus, A. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070509 · Cited by 61
The effects of naming style and expertise on program comprehension
Teasley, B. E. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1036 · Cited by 44
Extracting and analyzing time-series HCI data from screen-captured task videos
Bao, L., Li, J., Xing, Z., Wang, X., Xia, X., & Zhou, B. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9417-1 · Cited by 44
Self-citationTest confessions: A study of testing practices for plug-in systems
Greiler, M., Arie Van Deursen, A., & Storey, M. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227189 · Cited by 40
How Practitioners Perceive the Relevance of ESEM Research
Carver, J. C., Dieste, Ó., Kraft, N. A., Lo, D., & Zimmermann, T. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962597 · Cited by 31
Self-citationNavTracks: Supporting Navigation in Software
Singer, J., Elves, R., & Storey, M.-A. (2005) · IEEE International Workshop on Program Comprehension (IWPC) · doi:10.1109/wpc.2005.25 · Cited by 28
ActivitySpace: A Remembrance Framework to Support Interapplication Information Needs
Bao, L., Ye, D., Xing, Z., Xia, X., & Wang, X. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.90 · Cited by 22
Self-citationTracking and Analyzing Cross-Cutting Activities in Developers' Daily Work (N)
Bao, L., Xing, Z., Wang, X., & Zhou, B. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.43 · Cited by 15
Self-citation
Research works printed but not linked: 3
- whitaker, 1999, What causes IT workers to leave, Manag Rev, 88, 8
- spencer, 2009, Card Sorting Designing Usable Categories
- fjeldstad, 1983, Application program maintenance study: Report to our respondents, Proc GUIDE
Cited by
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2018) · Proceedings of the 26th Conference on Program Comprehension · doi:10.1145/3196321.3196334 · Cited by 621
Self-citationFew-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240
API method recommendation without worrying about the task-API knowledge gap
Huang, Q., Xia, X., Xing, Z., Lo, D., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238191 · Cited by 192
Self-citationHow do developers utilize source code from stack overflow?
Wu, Y., Wang, S., Bezemer, C.-P., & Inoue, K. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9634-5 · Cited by 127
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 148
Metadata
- Authors
- Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Singapore Management University (SG)
- Australian National University (AU)
- Queen's University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2018-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 951-976
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2741561716
- Cited by
- 326
- References
- 64
- Citations by year
- 2026: 30 · 2025: 37 · 2024: 54 · 2023: 62 · 2022: 60 · 2021: 38 · 2020: 22 · 2019: 14 · 2018: 8
- FWCI
- 22.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61602403, 61572426)
- National Key Technology R&D Program
- Ministry of Science and Technology of China (2015BAH17F01)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- program comprehension, professional developers, developer experience, project phases, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.