The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
When, how, and why developers (do not) test in their IDEs
Open accessAbstract
The research community in Software Engineering and Software Testing in particular builds many of its contributions on a set of mutually shared expectations. Despite the fact that they form the basis of many publications as well as open-source and commercial testing applications, these common expectations and beliefs are rarely ever questioned. For example, Frederic Brooks’ statement that testing takes half of the development time seems to have manifested itself within the community since he first made it in the “Mythical Man Month” in 1975. With this paper, we report on the surprising results of a large-scale field study with 416 software engineers whose development activity we closely monitored over the course of five months, resulting in over 13 years of recorded work time in their integrated development environments (IDEs). Our findings question several commonly shared assumptions and beliefs about testing and might be contributing factors to the observed bug proneness of software in practice: the majority of developers in our study does not test; developers rarely run their tests in the IDE; Test-Driven Development (TDD) is not widely practiced; and, last but not least, software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
Power-Law Distributions in Empirical Data
Clauset, A., Shalizi, C. R., & Newman, M. E. J. (2009) · SIAM Review · doi:10.1137/070710111 · Cited by 8,019
The Hawthorne effect: A reconsideration of the methodological artifact.
Adair, J. G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.2.334 · Cited by 1,095 · cited 1×
“The Hawthorne effect poses a similar threat [31]: participants of our study would be more prone to use, run and edit tests than they would do in general, because they know (1) that they are being measured and (2) they can preview a limited part of their behavior.”
Background(2014) · doi:10.1145/2597073.2597082 · Cited by 252
Self-citationMining StackOverflow to turn the IDE into a self-confident programming prompter
Ponzanelli, L., Bavota, G., Di Penta, M., Oliveto, R., & Lanza, M. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597077 · Cited by 240
Software developers' perceptions of productivity
Meyer, A. N., Fritz, T., Murphy, G. C., & Zimmermann, T. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635892 · Cited by 224 · cited 1×
“found that 47 out of 379 surveyed software engineering professionals perceive tasks such as testing as unproductive [39].”
Background
Cited by
A Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77
Razzaq, A., Buckley, J., Lai, Q., Yu, T., & Botterweck, G. (2025) · ACM Computing Surveys · doi:10.1145/3687299 · Cited by 36
Inference of development activities from interaction with uninstrumented applications
Bao, L., Xing, Z., Xia, X., Lo, D., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9547-8 · Cited by 25
Towards Activity-Aware Tool Support for Change Tasks
Kevic, K., & Fritz, T. (2017) · 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2017.48 · Cited by 5
Metadata
- Authors
- Beller, M., Gousios, G., Panichella, A., & Zaidman, A.
- Venue
- Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2015-08-30
- Pages
- 179-190
- OpenAlex
- W1997090065
- Cited by
- 162
- References
- 6
- FWCI
- 16.37
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When, how, and why developers (do not) test in their IDEs. https://app.cochat.ai/papers/10.1145/2786805.2786843
- The paper
- Beller, M., Gousios, G., Panichella, A., & Zaidman, A. (2015). When, how, and why developers (do not) test in their IDEs. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. https://doi.org/10.1145/2786805.2786843
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
