The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Evidence-based software engineering
Abstract
Our objective is to describe how software engineering might benefit from an evidence-based approach and to identify the potential difficulties associated with the approach. We compared the organisation and technical infrastructure supporting evidence-based medicine (EBM) with the situation in software engineering. We considered the impact that factors peculiar to software engineering (i.e. the skill factor and the lifecycle factor) would have on our ability to practice evidence-based software engineering (EBSE). EBSE promises a number of benefits by encouraging integration of research results with a view to supporting the needs of many different stakeholder groups. However, we do not currently have the infrastructure needed for widespread adoption of EBSE. The skill factor means software engineering experiments are vulnerable to subject and experimenter bias. The lifecycle factor means it is difficult to determine how technologies will behave once deployed. Software engineering would benefit from adopting what it can of the evidence approach provided that it deals with the specific problems that arise from the nature of software engineering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2002) · Cited by 13,211
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373
Preliminary guidelines for empirical research in software engineering
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., & Rosenberg, J. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1027796 · Cited by 1,510
Self-citationEffective Care in Pregnancy and Childbirth
Newton, E. (1990) · Journal of Human Lactation · doi:10.1177/089033449000600329 · Cited by 985
The Design of Replicated Studies
Lindsay, R. M., & Ehrenberg, A. S. C. (1993) · The American Statistician · doi:10.1080/00031305.1993.10475983 · Cited by 490
Review : Power and Information Technology Research: A Metatriangulation Review1, 2
Jasperson, J. (., Carte, T. A., Saunders, C. S., Butler, B. S., Croes, H. J. P., & Zheng, W. (2002) · MIS Quarterly · doi:10.2307/4132315 · Cited by 411
Towards an ontology of software maintenance
Kitchenham, B. A., Travassos, G. H., von Mayrhauser, A., Niessink, F., Schneidewind, N. F., Singer, J., Takada, S., Vehvilainen, R., & Yang, H. (1999) · Journal of Software Maintenance Research and Practice · doi:10.1002/(sici)1096-908x(199911/12)11:6<365::aid-smr200>3.0.co;2-w · Cited by 196
Self-citationPerspectives of Evidence-Based Surgery
Wente, M. N., Seiler, C. M., Uhl, W. H., & Büchler, M. W. (2003) · Digestive Surgery · doi:10.1159/000071183 · Cited by 173
Some questions about evidence-based practice in education
Hammersley, M. (2001) · Open Research Online - ORO (The Open University) · Cited by 149
Improve Software Quality by Reusing Knowledge and Experience
Basili, V. R., & Caldiera, G. (2005) · Springer eBooks · doi:10.1007/3-540-27662-9_16 · Cited by 137
Cited by
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234
Systematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239
“However, a general trend toward more evidence based software engineering (Kitchenham et al. 2004) has lead to an increased focus on new, empirical and systematic research methods.”
Recommended Steps for Thematic Synthesis in Software Engineering
Cruzes, D. S., & Dyba, T. (2011) · 2011 International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.36 · Cited by 960
Self-citationShahin, M., Ali Babar, M., & Zhu, L. (2017) · IEEE Access · doi:10.1109/access.2017.2685629 · Cited by 680
“EBSE Technical Report Ver.”
MethodInfluentialEvidence-based software engineering for practitioners
Dyba, T., Kitchenham, B. A., & Jorgensen, M. (2005) · IEEE Software · doi:10.1109/ms.2005.6 · Cited by 463
Self-citationAli, S., Briand, L. C., Hemmati, H., & Panesar-Walawege, R. K. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.52 · Cited by 452
Software development in startup companies: A systematic mapping study
Paternoster, N., Giardino, C., Unterkalmsteiner, M., Gorschek, T., & Abrahamsson, P. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.04.014 · Cited by 451
Applying Systematic Reviews to Diverse Study Types: An Experience Report
Dyba, T., Dingsoyr, T., & Hanssen, G. K. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.59 · Cited by 391
“Recent examples of such reviews include [9], [12], [16], [28], EBSE aims to improve decision making related to software development and maintenance by integrating current best evidence from research with practical experience and human values [19].”
BackgroundSelf-citationScientific research ontology to support systematic review in software engineering
de Almeida Biolchini, J. C., Mian, P. G., Natali, A. C. C., Conte, T. U., & Travassos, G. H. (2007) · Advanced Engineering Informatics · doi:10.1016/j.aei.2006.11.006 · Cited by 391
The Future of Empirical Methods in Software Engineering Research
Sjoberg, D. I. K., Dyba, T., & Jorgensen, M. (2007) · Future of Software Engineering (FOSE '07) · doi:10.1109/fose.2007.30 · Cited by 389
Self-citationResearch synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 346
A Visual Text Mining approach for Systematic Reviews
Malheiros, V., Hohn, E., Pinho, R., Mendonca, M., & Maldonado, J. C. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.21 · Cited by 329
Fernandes, M., Vieira, S. M., Leite, F., Palos, C., Finkelstein, S., & Sousa, J. M. C. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101762 · Cited by 312
Systematic reviews in sentiment analysis: a tertiary study
Ligthart, A., Catal, C., & Tekinerdogan, B. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09973-3 · Cited by 303
Reporting guidelines for controlled experiments in software engineering
Jedlitschka, A., & Pfahl, D. (2005) · 2005 International Symposium on Empirical Software Engineering, 2005. · doi:10.1109/isese.2005.1541818 · Cited by 295
Context in industrial software engineering research
Petersen, K., & Wohlin, C. (2009) · 2009 3rd International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316010 · Cited by 270
Local versus Global Lessons for Defect Prediction and Effort Estimation
Menzies, T., Butcher, A., Cok, D., Marcus, A., Layman, L., Shull, F., Turhan, B., & Zimmermann, T. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.83 · Cited by 261
A Map of Threats to Validity of Systematic Literature Reviews in Software Engineering
Zhou, X., Jin, Y., Zhang, H., Li, S., & Huang, X. (2016) · 2016 23rd Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2016.031 · Cited by 260
Developing search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200
Towards an Evidence-Based Understanding of Electronic Data Sources
Chen, L., Babar, M. A., & Zhang, H. (2010) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2010.17 · Cited by 191
“Secondary studies, such as Systematic Literature Reviews (SLR) and Systematic Mapping Studies (SMS), are the core methods of Evidence-Based Software Engineering (EBSE) [1], which is expected to help practitioners to make informed decisions about technology selection and adoption [2].”
How Reliable Are Systematic Reviews in Empirical Software Engineering?
MacDonell, S., Shepperd, M., Kitchenham, B., & Mendes, E. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.28 · Cited by 108
“We do not have a definitive answer to this question, but we note that Kitchenham et al. [5] analyzed the citations in each of the 10 primary studies they selected.”
InfluentialSelf-citationFinding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
“SE researchers have investigated various techniques, e.g. snowballing and pilot search, to increase the completeness of the selected relevant papers [19].”
BackgroundMethodSYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing
van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.685591 · Cited by 80
Investigating the Use of a Hybrid Search Strategy for Systematic Reviews
Mourao, E., Kalinowski, M., Murta, L., Mendes, E., & Wohlin, C. (2017) · 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem.2017.30 · Cited by 77
(2007) · doi:10.1049/iet-sen:20060052 · Cited by 67
Self-citationDescriptive theory of awareness for groupware development
Collazos, C. A., Gutiérrez, F. L., Gallardo, J., Ortega, M., Fardoun, H. M., & Molina, A. I. (2019) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-018-1165-9 · Cited by 65
A mapping study on documentation in Continuous Software Development
Theunissen, T., van Heesch, U., & Avgeriou, P. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2021.106733 · Cited by 40
Identifying the primary dimensions of DevSecOps: A multi-vocal literature review
Zhao, X., Clear, T., & Lal, R. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112063 · Cited by 40
Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews
Syriani, E., David, I., & Kumar, G. (2023) · arXiv · doi:10.48550/arxiv.2307.06464 · Cited by 27
Wyrich, M., Preikschat, A., Graziotin, D., & Wagner, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00055 · Cited by 24
Integrating security controls in DevSecOps: Challenges, solutions, and future research directions
Sinan, M., Shahin, M., & Gondal, I. (2025) · Journal of Software: Evolution and Process · doi:10.1002/smr.70029 · Cited by 13
Crossover Designs in Software Engineering Experiments: Review of the State of Analysis
Frattini, J., Fucci, D., & Vegas, S. (2024) · Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690754 · Cited by 3
Metadata
- Authors
- Kitchenham, B. A., Dyba, T., & Jorgensen, M.
- Venue
- Proceedings. 26th International Conference on Software Engineering
- Publisher
- IEEE Comput. Soc
- Published
- 2004-09-28
- Pages
- 273-281
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4247781538
- Cited by
- 455
- References
- 14
- FWCI
- 10.32
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.