Understanding replication of experiments in software engineering: A classification
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Abstract Context Replication plays an important role in experimental disciplines. There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective To improve our understanding of SE experiment replication, in this work we propose a classification which is intend to provide experimenters with guidance about what types of replication they can perform. Method The research approach followed is structured according to the following activities: (1) a literature review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion The aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · cited 9×
“replication AND kind Lüdtke [44] Replication W/2 type Schmidt [61] Radder [56] U”
BackgroundMethodInfluentialMost published research findings are false-but a little replication goes a long way.
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040028 · Cited by 341 · cited 1×
“Replication is at the heart of the experimental paradigm [61] and is considered to be the cornerstone of scientific knowledge [53].”
BackgroundDeveloping search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200 · cited 1×
“’s advice [19]: it covers a wide range of publications in different branches of science, and, the search engine is very robust.”
BackgroundSelf-citationReproducibility: The risks of the replication drive
Bissell, M. (2013) · Nature · doi:10.1038/503333a · Cited by 99 · cited 2×
“– Bisell [8] claims that it takes her from four to six years, and at times much longer, to get results conclusive enough to be able to write a paper about her experiments on the roles of the microenvironment and extracellular matrix in cancer.”
BackgroundResultReplicating software engineering experiments: a poisoned chalice or the Holy Grail
Miller, J. (2005) · Information and Software Technology · doi:10.1016/j.infsof.2004.08.005 · Cited by 84 · cited 5×
“Thus, our classification is capable of accounting for replications as far apart as Basili et al.’s strict replications [5] or replications that retain no more than the hypothesis, which Kitchenham [38] or Miller [51] suggest are the only replications that are of any use.”
BackgroundInfluential
Cited by
Does ACM’s code of ethics change ethical decision making in software development?
McNamara, A., Smith, J., & Murphy-Hill, E. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264833 · Cited by 244
“To answer this question, we performed a conceptual replication (or a reproduction, according to Gómez and colleagues’ classification [13]) of a study by Cleek and Leonard [8].”
MethodCrossover Designs in Software Engineering Experiments: Benefits and Perils
Vegas, S., Apa, C., & Juristo, N. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2467378 · Cited by 145
Self-citationThe role and value of replication in empirical software engineering results
Shepperd, M., Ajienka, N., & Counsell, S. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.01.006 · Cited by 60
A family of experiments on test-driven development
Santos, A., Vegas, S., Dieste, O., Uyaguari, F., Tosun, A., Fucci, D., Turhan, B., Scanniello, G., Romano, S., Karac, I., et al. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09895-8 · Cited by 30
Self-citation
Metadata
- Authors
- Gómez, O. S., Juristo, N., & Vegas, S.
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2014-08
- Volume
- 56
- Issue
- 8
- Pages
- 1033-1048
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W2064161267
- Cited by
- 125
- References
- 5
- FWCI
- 28.66
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.