Developer Testing in the IDE: Patterns, Beliefs, and Behavior
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Software testing is one of the key activities to achieve software quality in practice. Despite its importance, however, we have a remarkable lack of knowledge on how developers test in real-world projects. In this paper, we report on a large-scale field study with 2,443 software engineers whose development activities we closely monitored over 2.5 years in four integrated development environments (IDEs). Our findings, which largely generalized across the studied IDEs and programming languages Java and C#, question several commonly shared assumptions and beliefs about developer testing: half of the developers in our study do not test; developers rarely run their tests in the IDE; most programming sessions end without any test execution; only once they start testing, do they do it extensively; a quarter of test cases is responsible for three quarters of all test failures; 12 percent of tests show flaky behavior; Test-Driven Development (TDD) is not widely practiced; and software developers only spend a quarter of their time engineering tests, whereas they think they test half of their time. We summarize these practices of loosely guiding one's development efforts with the help of testing in an initial summary on Test-Guided Development (TGD), a behavior we argue to be closer to the development reality of most developers than TDD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
An Analysis of Variance Test for Normality (Complete Samples)
Shapiro, S. S., & Wilk, M. B. (1965) · Biometrika · doi:10.2307/2333709 · Cited by 23,469
(1975) · doi:10.1145/800027.808439 · Cited by 3,227
The Hawthorne effect: A reconsideration of the methodological artifact.
Adair, J. G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.2.334 · Cited by 1,095 · cited 1×
“The Hawthorne effect [89] poses a similar threat: participants of our studywould bemore prone to use, run, and edit”
Background(2014) · doi:10.1145/2597073.2597082 · Cited by 252 · cited 1×
“their untested changes to the Gerrit review tool [50] and rely on it to trigger the execution of the tests on the CI server.”
MethodSelf-citationMining StackOverflow to turn the IDE into a self-confident programming prompter
Ponzanelli, L., Bavota, G., Di Penta, M., Oliveto, R., & Lanza, M. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597077 · Cited by 240 · cited 1×
“Researchers try to integrate answers from internet fora such as Stack Overflow into the IDE [60], to make this possibly interrupting context switch unnecessary.”
BackgroundSoftware developers' perceptions of productivity
Meyer, A. N., Fritz, T., Murphy, G. C., & Zimmermann, T. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635892 · Cited by 224 · cited 1×
“found similar numbers when presenting developers with a report of their productivity [90].”
BackgroundCode churn: a measure for estimating the impact of code change
Munson, J. C., & Elbaum, S. G. (2002) · Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272) · doi:10.1109/icsm.1998.738486 · Cited by 153
A Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77
Self-citation
Cited by
Today was a good day: The daily life of software developers
Meyer, A. N., Barr, E. T., Bird, C., & Zimmermann, T. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2904957 · Cited by 123
Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity
Beller, M., Orgovan, V., Buja, S., & Zimmermann, T. (2021) · IEEE Software · doi:10.1109/ms.2020.3048200 · Cited by 16
Self-citationStudying test-driven development and its retainment over a six-month time span
Baldassarre, M. T., Caivano, D., Fucci, D., Juristo, N., Romano, S., Scanniello, G., & Turhan, B. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110937 · Cited by 15
Affective reactions and test-driven development: Results from three experiments and a survey
Baldassarre, M. T., Caivano, D., Fucci, D., Romano, S., & Scanniello, G. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111154 · Cited by 9
Metadata
- Authors
- Beller, M., Gousios, G., Panichella, A., Proksch, S., Amann, S., & Zaidman, A.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2019-03-01
- Volume
- 45
- Issue
- 3
- Pages
- 261-284
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2770290320
- Cited by
- 101
- References
- 9
- FWCI
- 14.77
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Developer Testing in the IDE: Patterns, Beliefs, and Behavior. https://app.cochat.ai/papers/10.1109/tse.2017.2776152
- The paper
- Beller, M., Gousios, G., Panichella, A., Proksch, S., Amann, S., & Zaidman, A. (2019). Developer Testing in the IDE: Patterns, Beliefs, and Behavior. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2017.2776152
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.