The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
A Systematic Review of Theory Use in Software Engineering Experiments
Abstract
Empirically based theories are generally perceived as foundational to science. However, in many disciplines, the nature, role and even the necessity of theories remain matters for debate, particularly in young or practical disciplines such as software engineering. This article reports a systematic review of the explicit use of theory in a comprehensive set of 103 articles reporting experiments, from of a total of 5,453 articles published in major software engineering journals and conferences in the decade 1993-2002. Of the 103 articles, 24 use a total of 40 theories in various ways to explain the cause-effect relationship(s) under investigation. The majority of these use theory in the experimental design to justify research questions and hypotheses, some use theory to provide post hoc explanations of their results, and a few test or modify theory. A third of the theories are proposed by authors of the reviewed articles. The interdisciplinary nature of the theories used is greater than that of research in software engineering in general. We found that theory use and awareness of theoretical issues are present, but that theory-driven research is, as yet, not a major issue in empirical software engineering. Several articles comment explicitly on the lack of relevant theory. We call for an increased awareness of the potential benefits of involving theory, when feasible. To support software engineering researchers who wish to use theory, we show which of the reviewed articles on which topics use which theories for what purposes, as well as details of the theories' characteristics
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587
Organizational Information Requirements, Media Richness and Structural Design
Daft, R. L., & Lengel, R. H. (1986) · Management Science · doi:10.1287/mnsc.32.5.554 · Cited by 9,868
A spreading-activation theory of semantic processing.
Collins, A. M., & Loftus, E. F. (1975) · Psychological Review · doi:10.1037/0033-295x.82.6.407 · Cited by 8,141
Human Memory: A Proposed System and its Control Processes
Atkinson, R. C., & Shiffrin, R. M. (1968) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60422-3 · Cited by 7,216
Fraassen, B. C. V. (1980) · doi:10.1093/0198244274.001.0001 · Cited by 4,682
(2007) · Cognitive Psychology · doi:10.4135/9781446212967.n15 · Cited by 3,749
The Nature of Theory in Information Systems1
Gregor, S. (2006) · MIS Quarterly · doi:10.2307/25148742 · Cited by 3,306 · cited 6×
“All these issues have a bearing on the nature and components of prospective theories.”
BackgroundMethodInfluentialDuncker, K. (1945) · Psychological Monographs · doi:10.1037/h0093599 · Cited by 2,938
Cited by
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
“[12] 2007 Research trends Theory in SE experiments SLR Guideline TR No 103”
BackgroundGuidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862
“Communicating research results in terms of theories is an underdeveloped practice in software engineering (Hannay et al. 2007).”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239
“The high number of categories for systematic reviews and their detail level is particularly visible for the reviews (Hannay et al. 2007, Sjøberg et al. 2005).”
Using mapping studies as the basis for further research – A participant-observer case study
Kitchenham, B. A., Budgen, D., & Pearl Brereton, O. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.011 · Cited by 621
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
Applying Systematic Reviews to Diverse Study Types: An Experience Report
Dyba, T., Dingsoyr, T., & Hanssen, G. K. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.59 · Cited by 391
The Future of Empirical Methods in Software Engineering Research
Sjoberg, D. I. K., Dyba, T., & Jorgensen, M. (2007) · Future of Software Engineering (FOSE '07) · doi:10.1109/fose.2007.30 · Cited by 389
Self-citationResearch synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 346
A Visual Text Mining approach for Systematic Reviews
Malheiros, V., Hohn, E., Pinho, R., Mendonca, M., & Maldonado, J. C. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.21 · Cited by 329
A method for evaluating rigor and industrial relevance of technology evaluations
Ivarsson, M., & Gorschek, T. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9146-4 · Cited by 279
Software process improvement in small and medium software enterprises: a systematic review
Pino, F. J., García, F., & Piattini, M. (2008) · Software Quality Journal · doi:10.1007/s11219-007-9038-z · Cited by 274
(2015) · doi:10.1007/s10664-013-9279-3 · Cited by 273
The effectiveness of pair programming: A meta-analysis
Hannay, J. E., Dybå, T., Arisholm, E., & Sjøberg, D. I. K. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.02.001 · Cited by 268
Self-citationStrength of evidence in systematic reviews in software engineering
Dybå, T., & Dingsøyr, T. (2008) · Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/1414004.1414034 · Cited by 261
Developing search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200
The daily stand-up meeting: A grounded theory study
Stray, V., Sjøberg, D. I. K., & Dybå, T. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.01.004 · Cited by 139
Self-citationSystematic literature reviews in software engineering
Wohlin, C., & Prikladnicki, R. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.002 · Cited by 120
An Information Foraging Theory Perspective on Tools for Debugging, Refactoring, and Reuse Tasks
Fleming, S. D., Scaffidi, C., Piorkowski, D., Burnett, M., Bellamy, R., Lawrance, J., & Kwan, I. (2013) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2430545.2430551 · Cited by 79
Theory-oriented software engineering
Stol, K.-J., & Fitzgerald, B. (2015) · Science of Computer Programming · doi:10.1016/j.scico.2014.11.010 · Cited by 63
Al-Sakkaf, A., Omar, M., & Ahmad, M. (2019) · Computer Science Education · doi:10.1080/08993408.2018.1564611 · Cited by 19
Metadata
- Authors
- Hannay, J. E., Sjoberg, D. I. K., & Dyba, T.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2007-02
- Volume
- 33
- Issue
- 2
- Pages
- 87-107
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2122626138
- Cited by
- 226
- References
- 24
- FWCI
- 7.65
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.