A family of experiments on test-driven development
Open accessThe DOI matches its Crossref record; 55 of the 57 references it cites were found in a registry or an index.
Abstract
Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable. The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD. We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD. TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students. Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process, or case studies comparing the performance achieved by TDD vs. the control approach (e.g., the waterfall model), each applied to develop a different system. Further experiments with TDD experts are needed to validate these hypotheses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · cited 1×
“6 Throughout the rest of the paper we refer to the treatment—terminology commonly used in experimental design and data analysis [11,23,26,58]—and the development approach (i.”
Method(2017) · Cited by 16,319
Experimental Design and Data Analysis for Biologists
Quinn, G. P., & Keough, M. J. (2002) · doi:10.1017/cbo9780511806384 · Cited by 10,674 · cited 2×
“, points with abnormally high X values—in our case, reported experience levels) that pull the regression lines towards each other, especially if sample size is small [41].”
BackgroundKruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
Likert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“This approach is typically followed in other disciplines (Norman 2010).”
BackgroundMeta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460 · cited 5×
“The research method according to Wohlin et. al.’s definitions (Wohlin et al. 2012): – Surveys: empirical studies that collect information from or about people to describe, compare or explain their knowledge, attitudes and behavior For us, the defining characteristic of surveys is that their results…”
BackgroundMethodInfluentialMeta-Analytic Procedures for Social Research.
Metcalfe, A., & Rosenthal, R. (1994) · The Statistician · doi:10.2307/2348600 · Cited by 4,249
(2011) · Cited by 2,025
The Importance of the Normality Assumption in Large Public Health Data Sets
Lumley, T., Diehr, P., Emerson, S., & Chen, L. (2002) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.23.100901.140546 · Cited by 1,803
Research works printed but not linked: 5
- Borenstein M, Hedges L V, Higgins JPT, Rothstein H R (2011) Introduction to meta-analysis. Wiley
- Kollanus S (2010) Test-driven development-still a promising approach? In: Quality of Information and Communications Technology (QUATIC), 2010 Seventh International Conference on the. , pp 403–408
- Martin CR (2001) Advanced principles, patterns and process of software development. Prentice Hall
- Santos A, Gomez O S, Juristo N (2018a) Analyzing families of experiments in SE: a systematic mapping study. IEEE Trans Softw Eng:1. https://doi.org/10.1109/TSE.2018.2864633 · DOI
- Santos A, Vegas S, Oivo M, Juristo N (2018c) Guidelines for analyzing families of experiments in SE. Submitted to IEEE Transactions on Software Engineering
Cited by
How Developers Engineer Test Cases: An Observational Study
Aniche, M. F., Treude, C., & Zaidman, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3129889 · Cited by 83
“Test-driven development (TDD) is an agile software development approach in which software developed in small iterative testing-coding cycles [61].”
MethodUsing the SOCIO Chatbot for UML Modelling: A Family of Experiments
Ren, R., Castro, J. W., Santos, A., Dieste, Ó., & Acuña, S. T. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3150720 · Cited by 22
“We acknowledge that, although the experimental material was translated into the participants’ native language to make them feel comfortable, the self-assessment questions may not accurately capture their background [64], [65].”
BackgroundSelf-citationFuzzing microservices: A series of user studies in industry on industrial systems with EvoMaster
Zhang, M., Arcuri, A., Li, Y., Liu, Y., Xue, K., Wang, Z., Huo, J., & Huang, W. (2025) · Science of Computer Programming · doi:10.1016/j.scico.2025.103322 · Cited by 31
“Five ‘‘toy tasks’’ [62] were used for the experiments.”
Gamification in Test-Driven Development Practice
Ren, W., & BARRETT, S. (2023) · doi:10.1145/3617553.3617889 · Cited by 17
“Additionally, [51] found a strong correlation between TDD behaviors and software maintain-ability.”
BackgroundFerreira, J. M., Rodríguez, F. D., Santos, A., Dieste, Ó., Acuña, S. T., & Juristo, N. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3149586 · Cited by 36
“Another family of experiments evaluated whether the use of test-driven development (TDD) improves software product quality [50].”
BackgroundSelf-citation
Metadata
- Authors
- Santos, A., Vegas, S., Dieste, O., Uyaguari, F., Tosun, A., Fucci, D., Turhan, B., Scanniello, G., Romano, S., Karac, I., Kuhrmann, M., Mandić, V., Ramač, R., Pfahl, D., Engblom, C., Kyykka, J., Rungi, K., Palomeque, C., Spisak, J., Oivo, M., & Juristo, N.
- Affiliations
- Universidad Politécnica de Madrid (ES)
- Istanbul Technical University (TR)
- Blekinge Institute of Technology (SE)
- University of Oulu (FI)
- University of Basilicata (IT)
- University of Bari Aldo Moro (IT)
- University of Passau (DE)
- University of Novi Sad (RS)
- University of Tartu (EE)
- Ericsson (Finland) (FI)
- Corresponding author
- Vegas, Sira
- ORCID
- Venue
- Empirical Software Engineering
- Published
- 2021-03-23
- Volume
- 26
- Issue
- 3
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W3109605734
- Cited by
- 30
- References
- 57
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 8 · 2023: 7 · 2022: 3 · 2021: 4
- FWCI
- 4.97
- Type
- journal article
- Language
- en
- Open access
- green
- oa.upm.es
- Licence
- cc-by
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Empirical Software Engineering (publishedVersion)
- Hispana (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Archivo Digital UPM (Universidad Politécnica de Madrid) (submittedVersion, cc-by)
- Reutlingen University Academic Bibliography (Reutlingen University) (submittedVersion)
- Istanbul Technical University Academic Open Archive (Istanbul Technical University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ministerio de Ciencia, Innovación y Universidades (PGC2018-097265-B-I00)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- test-driven development, code quality, learning effects, agile software development, software quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A family of experiments on test-driven development. https://app.cochat.ai/papers/10.1007/s10664-020-09895-8
- The paper
- Santos, A., Vegas, S., Dieste, O., Uyaguari, F., Tosun, A., Fucci, D., Turhan, B., Scanniello, G., Romano, S., Karac, I., Kuhrmann, M., Mandić, V., Ramač, R., Pfahl, D., Engblom, C., Kyykka, J., Rungi, K., Palomeque, C., Spisak, J., . . . Juristo, N. (2021). A family of experiments on test-driven development. Empirical Software Engineering. https://doi.org/10.1007/s10664-020-09895-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.