Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity
Open accessAbstract
Two separate schools of thought to measure software engineers' productivity have coexisted: for a long time: on the one hand, using automated product or process measures and, on the other hand, using self-reported or perceived productivity. In this article, we start to bridge the gap between them with an empirical study of 81 software developers at Microsoft.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A more holistic picture of productivity is formed by measuring perceived productivity through self-reported data across various dimensions and supplementing it with automatically measured data.
“A more holistic picture is formed by measuring perceived productivity through self-reported data across various dimensions [6], and supplementing it with automatically measured data [4].”
From Productivity Assessment of Neural Code Completion (2022) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Measuring application development productivity
Albrecht, A. (1979) · Medical Entomology and Zoology · Cited by 1,119 · see citing works
The Work Life of Developers: Activities, Switches and Perceived Productivity
Meyer, A. N., Barton, L. E., Murphy, G. C., Zimmermann, T., & Fritz, T. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2656886 · Cited by 170 · cited 2× · see citing works
“3) From previous studies on productivity we know that rating behavior differs tremendously between individuals [6], [9], [10].”
Self-citationMeasuring and predicting software productivity: A systematic map and review
Petersen, K. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.001 · Cited by 128 · cited 1× · see citing works
“Two main methods on assessing productivity have thus far emerged: 1) The automated measurement of product or process features to assess productivity, such as the lines of code written per day [3], sometimes called traditional”
Today was a good day: The daily life of software developers
Meyer, A. N., Barr, E. T., Bird, C., & Zimmermann, T. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2904957 · Cited by 123 · cited 1× · see citing works
“Separate research found that these pre-defined attributes distinguish a good from a bad day for a developer [8], such as”
Self-citationDeveloper Testing in the IDE: Patterns, Beliefs, and Behavior
Beller, M., Gousios, G., Panichella, A., Proksch, S., Amann, S., & Zaidman, A. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2776152 · Cited by 103 · see citing works
Self-citationWhat Predicts Software Developers’ Productivity?
Murphy-Hill, E., Jaspan, C., Sadowski, C., Shepherd, D., Phillips, M., Winter, C., Knight, A., Smith, E., & Jorde, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2900308 · Cited by 100 · see citing works
Characterizing Software Engineering Work with Personas Based on Knowledge Worker Actions
Ford, D., Zimmermann, T., Bird, C., & Nagappan, N. (2017) · 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem.2017.54 · Cited by 37 · see citing works
Self-citationCharacterizing Software Developers by Perceptions of Productivity
Meyer, A. N., Zimmermann, T., & Fritz, T. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.17 · Cited by 36 · cited 2× · see citing works
“3) From previous studies on productivity we know that rating behavior differs tremendously between individuals [6], [9], [10].”
Self-citationTowards actionable visualization for software developers
Merino, L., Ghafari, M., & Nierstrasz, O. (2017) · Journal of Software Evolution and Process · doi:10.1002/smr.1923 · Cited by 12 · see citing works
Flagged first, then by citations
1 of the 10 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [1]1. S. Wagner and M. Ruhe: A Systematic Review of Produc- tivity Factors in Software Development, Preprint, 2018
- [4]4. A.J. Albrecht: Measuring Application Development Pro- ductivity, Proc. of IBM Application Development Sympo- sium, 1979
Cited by
Sorted by citations
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200 · see citing works
“A more holistic picture is formed by measuring perceived productivity through self-reported data across various dimensions [6], and supplementing it with automatically measured data [4].”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · see citing works
“A more holistic picture is formed by measuring perceived productivity through self-reported data across various dimensions [6], and supplementing it with automatically measured data [4].”
BackgroundThe SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164 · see citing works
Self-citationMeasuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · see citing works
“4 We used the SPACE framework 6 to design a survey that captures self-reported productivity and paired the self-reported data with usage telemetry.”
MethodEmotions and Perceived Productivity of Software Developers at the Workplace
Girardi, D., Lanubile, F., Novielli, N., & Serebrenik, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087906 · Cited by 95 · see citing works
“Overall, our results suggest that emotions might act as a proxy for productivity, towards bridging automated measures and self-report for productivity assessment [62].”
BackgroundThe SPACE of developer productivity
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Communications of the ACM · doi:10.1145/3453928 · Cited by 21 · see citing works
“Let’s examine the SPACE framework in more depth by looking further at the dimensions of (1) activity and (2) efficiency and flow.”
BackgroundSelf-citationStress, motivation, and performance in global software engineering
Suárez-Albanchez, J., & Vizcaíno, A. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2600 · Cited by 19 · see citing works
“42 For instance, ˇ Smite et al. 14 studied how home telework affected productivity during the Covid-19 pandemic.”
BackgroundToward Effective AI Support for Developers
Khemka, M., & Houck, B. (2024) · Communications of the ACM · doi:10.1145/3690928 · Cited by 18 · see citing works
Software Productivity in Practice: A Systematic Mapping Study
Duarte, C. (2022) · Softw. · doi:10.3390/software1020008 · Cited by 10 · see citing works
“The included papers report on experiments [22,23] and case studies [71,84,97] that investigate specific productivity factors, discuss the adequacy of regression models for model-based development productivity prediction [86] and selfreported productivity evaluation [74].”
BackgroundMethodInfluentialBreaking the Flow: A Study of Interruptions During Software Engineering Activities
Ma, Y., Huang, Y., & Leach, K. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639079 · Cited by 12 · see citing works
Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era
Kumar, S., Goel, D., Zimmermann, T., Houck, B., Ashok, B., & Bansal, C. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00007 · Cited by 8 · see citing works
“More recent works have captured a more holistic picture of what a developer does and aimed to capture the time spent through various methods of capturing data like surveys [3], [15], [16], observations and interviews [17], and through tracking usage of the computer [13], [14].”
Self-citationToward Effective AI Support for Developers
Khemka, M., & Houck, B. (2024) · Queue · doi:10.1145/3675416 · Cited by 5 · see citing works
Understanding Remote Work Experience: Insights Into Well‐Being
Vizcaíno, A., Suárez-Albanchez, J., Šmite, D., & García, F. O. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.2757 · Cited by 4 · see citing works
“In software engineering, it is still a topical issue that is continuously studied owing to the rapid evolution of methodologies and working conditions [34–37].”
Usage patterns of software product metrics in assessing developers’ output: A comprehensive study
Chen, W., Yu, H., Fan, G., Huang, Z., & Liang, Y. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107935 · Cited by 4 · see citing works
Understanding and Measuring Perceived Productivity
Gennara, A. (2023) · doi:10.22215/etd/2023-15741 · Cited by 1 · see citing works
Wivestad, V. T., & Ulfsnes, R. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_5
Am I Productive? Exploring the Experience of Remote Workerswith Task Management Tools
Beale, R., & Williams, O. (2025) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3750069.3750107
“The SPACE framework [6] proposes a more robust productivity system of measurement, including collaboration.”
- Authors
- Beller, M., Orgovan, V., Buja, S., & Zimmermann, T.
- Affiliations
- Meta (United States) (US)
- Microsoft (United States) (US)
- Venue
- IEEE Software
- Published
- 2021-09
- Volume
- 38
- Issue
- 5
- Pages
- 24-31
- ISSN
- 0740-7459 (ISSN-L)
- 1937-4194
- OpenAlex
- W3111932144
- Cited by
- 16
- References
- 10
- Citations by year
- 2025: 5 · 2024: 4 · 2023: 2 · 2022: 2 · 2021: 3
- FWCI
- 1.65
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Software (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
- Keywords
- software productivity measurement, software developers, developer productivity, perceived productivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
