The Role of Deliberate Artificial Design Elements in Software Engineering Experiments
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Increased realism in software engineering experiments is often promoted as an important means to increase generalizability and industrial relevance. In this context, artificiality, e.g., the use of constructed tasks in place of realistic tasks, is seen as a threat. In this article, we examine the opposite view, that deliberately introduced artificial design elements may increase knowledge gain and enhance both generalizability and relevance. In the first part of the article, we identify and evaluate arguments and examples in favor of, and against, deliberately introducing artificiality into software engineering experiments. In the second part of the article, we summarize a content analysis of articles reporting software engineering experiments published over the ten-year period 1993-2002. The analysis reveals a striving for realism and external validity, but little awareness of for what and when, various degrees of artificiality and realism are appropriate. We conclude that an increased awareness and deliberation in these respects is essential. However, arguments in favor of artificial design elements should not be used to justify studies that are badly designed or that have research questions of low relevance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Anomalies: The Endowment Effect, Loss Aversion, and Status Quo Bias
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1991) · Journal of Economic Perspectives · doi:10.1257/jep.5.1.193 · Cited by 5,019
Fraassen, B. C. V. (1980) · doi:10.1093/0198244274.001.0001 · Cited by 4,682
Organizational Theories: Some Criteria for Evaluation
Bacharach, S. B. (1989) · The Academy of Management Review · doi:10.5465/amr.1989.4308374 · Cited by 2,305
Cognitive Fit: An Empirical Study of Information Acquisition
Vessey, I., & Galletta, D. (1991) · Information Systems Research · doi:10.1287/isre.2.1.63 · Cited by 705 · cited 1×
“In order to investigate the theory of cognitive fit [103], [3], a real-world problem was made more data intensive (and, hence, less process intensive) so as to act as a match for a data-centered methodology in comparison to a process-centered methodology.”
ResultReplicating software engineering experiments: a poisoned chalice or the Holy Grail
Miller, J. (2005) · Information and Software Technology · doi:10.1016/j.infsof.2004.08.005 · Cited by 84 · cited 1×
“It is vital that experiments in empirical software engineering do not become static repetitions of each other on only one level of inquiry (see also [76]).”
BackgroundThe Scope of General Theory: Methods for Linking Deductive and Inductive Comparative History
BOSWELL, T., & BROWN, C. (1999) · Sociological Methods & Research · doi:10.1177/0049124199028002002 · Cited by 45 · cited 1×
“…states the conditions under which various consequences (possibly disparate but related) may be derived from the knowledge (see, e.g., [16], [73]). from this information " [12] (emphasis added), and Hö st et al. [51] conclude that researchers need to capture underlying explanatory…”
Background
Cited by
Systematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230
“or inspired by, or informed by the Guidelines or used criteria for inclusion exclusion as suggested by the Guidelines [10,19,20,11].”
BackgroundMethodInfluentialUsing mapping studies as the basis for further research – A participant-observer case study
Kitchenham, B. A., Budgen, D., & Pearl Brereton, O. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.011 · Cited by 621
Research synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 346
The effectiveness of pair programming: A meta-analysis
Hannay, J. E., Dybå, T., Arisholm, E., & Sjøberg, D. I. K. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.02.001 · Cited by 268
Self-citationThe ABC of Software Engineering Research
Stol, K.-J., & Fitzgerald, B. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3241743 · Cited by 245
Metadata
- Authors
- Hannay, J. E., & Jorgensen, M.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2008-03
- Volume
- 34
- Issue
- 2
- Pages
- 242-259
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W1985558141
- Cited by
- 55
- References
- 7
- FWCI
- 16.58
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.