Guidelines for conducting and reporting case study research in software engineering
Open accessThe DOI matches its Crossref record; 74 of the 75 references it cites were found in a registry or an index.
Abstract
Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors’ own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it is common to include construct validity, reliability, internal validity, and external validity, according to the guideline proposed in other works [222,255], which we adopted in the following.”
From Sustainability in computing education: A systematic literature review (2024) · cites this paper as [222]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1984) · Cited by 81,983
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,535
Basics of Qualitative Research
Ozanne, J. L., Strauss, A., & Corbin, J. (1992) · Journal of Marketing Research · doi:10.1177/002224379202900312 · Cited by 17,866
Basics of Qualitative Research
Ozanne, J. L., Strauss, A., & Corbin, J. (1992) · Journal of Marketing Research · doi:10.2307/3172751 · Cited by 17,850
Building Theories from Case Study Research
Eisenhardt, K. M. (1989) · The Academy of Management Review · doi:10.2307/258557 · Cited by 17,455
The Art of Case Study Research
Johnson, K. E., & Stake, R. E. (1996) · The Modern Language Journal · doi:10.2307/329758 · Cited by 15,826 · cited 4×
“Four different types of triangulation may be applied ( Stake 1995 ):”
BackgroundMethodInfluentialHartley, J. (2004) · Essential Guide to Qualitative Methods in Organizational Research · doi:10.4135/9781446280119.n26 · Cited by 15,461
Five misunderstandings about case-study research
Flyvbjerg, B. (2006) · Qualitative Inquiry · doi:10.1177/1077800405284363 · Cited by 14,266 · cited 4×
“2 Case study methodology was originally used primarily for exploratory purposes, and some researchers still limit case studies for this purpose, as discussed by Flyvbjerg (2007).”
BackgroundMethodInfluentialAn Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263 · cited 7×
“We prefer to consider ethnographic studies as a specialized type of case studies with focus on cultural practices (Easterbrook et al. 2008) or long duration studies with large amounts of participant-observer data (Klein and Myers 1999).”
BackgroundMethodInfluentialThe Case Research Strategy in Studies of Information Systems
Benbasat, I., Goldstein, D. K., & Mead, M. (1987) · MIS Quarterly · doi:10.2307/248684 · Cited by 4,716 · cited 15×
“The purpose of the selection may be to study a case that is expected to be “ typical ” , “ critical ” , “ revelatory ” or “ unique ” in some respect (Benbasat et al. 1987), and the case is selected accordingly.”
BackgroundMethodInfluentialSoftware Metrics: A Rigorous and Practical Approach
Fenton, N., & Pfleeger, S. L. (2013) · Cited by 2,509
Experimentation in Software Engineering: An Introduction
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2011) · Lund University Publications (Lund University) · Cited by 2,214
Self-citationExperimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000) · The Kluwer International Series in Software Engineering · doi:10.1007/978-1-4615-4625-2 · Cited by 1,776
Self-citationEmpirical research methods in operations management
Flynn, B. B., Sakakibara, S., Schroeder, R. G., Bates, K. A., & Flynn, E. J. (1990) · Journal of Operations Management · doi:10.1016/0272-6963(90)90098-x · Cited by 1,634
Preliminary guidelines for empirical research in software engineering
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., & Rosenberg, J. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1027796 · Cited by 1,510 · cited 4×
“There exist guidelines for experiments ’ conduct (Kitchenham et al. 2002; Wohlin et al. 2000) and reporting (Jedlitschka and Pfahl 2005), measurements (Basili and Weiss 1984; Fenton and Pfleeger 1996; van Solingen and Berghout 1999), and systematic reviews (Kitchenham 2007), while only little is…”
InfluentialQualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493
Five Misunderstandings About Case-Study Research
Flyvbjerg, B., & Flyvbjerg, B. (2016) · Case Studies · doi:10.4135/9781473915480.n40 · Cited by 1,414
A Scientific Methodology for MIS Case Studies
Lee, A. S. (1989) · MIS Quarterly · doi:10.2307/248698 · Cited by 1,300 · cited 5×
“is close to the natural science research model (Lee 1989) and related to Robson’s explanatory category.”
BackgroundInfluentialSelecting Empirical Methods for Software Engineering Research
Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_11 · Cited by 1,167
A Methodology for Collecting Valid Software Engineering Data
Basili, V. R., & Weiss, D. M. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010301 · Cited by 1,149 · cited 4×
“The definition of what data to collect should be based on a goal-oriented measurement technique, such as the Goal Question Metric method (GQM) (Basili and Weiss 1984; van Solingen and Berghout 1999).”
BackgroundMethodInfluentialGuide to Advanced Empirical Software Engineering
(2008) · Springer eBooks · doi:10.1007/978-1-84800-044-5 · Cited by 1,003
A survey of controlled experiments in software engineering
Sjoeberg, D. I. K., Hannay, J. E., Hansen, O., Kampenes, V. B., Karahasanovic, A., Liborg, N.-K., & Rekdal, A. C. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.97 · Cited by 682 · cited 4×
“This is worth a study on its own, similar to the systematic review on experiments by Sjøberg et al. (2005).”
BackgroundMethodResultInfluentialCase studies for method and tool evaluation
Kitchenham, B., Pickard, L., & Pfleeger, S. L. (1995) · IEEE Software · doi:10.1109/52.391832 · Cited by 601 · cited 2×
“2000) and reporting (Jedlitschka and Pfahl 2005), measurements (Basili and Weiss 1984; Fenton and Pfleeger 1996; van Solingen and Berghout 1999), and systematic reviews (Kitchenham 2007), while only little is written on case studies in software engineering (Höst and Runeson 2007; Kitchenham et al. 1995; Wohlin et al.…”
BackgroundExperimentation in software engineering
Basili, V. R., Selby, R. W., & Hutchens, D. H. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312975 · Cited by 553
Experimental models for validating technology
Zelkowitz, M. V., & Wallace, D. R. (1998) · Computer · doi:10.1109/2.675630 · Cited by 524 · cited 4×
“Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study and field study as observational methods (Zelkowitz and Wallace 1998).”
BackgroundInfluentialStudying Software Engineers: Data Collection Techniques for Software Field Studies
Lethbridge, T. C., Sim, S. E., & Singer, J. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-1290-x · Cited by 494
Should computer scientists experiment more?
Tichy, W. F. (1998) · Computer · doi:10.1109/2.675631 · Cited by 487 · cited 2×
“Case study methodology handbooks are superfluously available in e.g. social sciences (Robson 2002; Stake 1995; Yin 2003) which literature also has been used in software engineering.”
BackgroundMethodThe Goal/Question/Metric Method : A Practical Guide for Quality Improvement of Software Development
Berghout, E. W., & van Solingen, R. (1998) · Cited by 483
A Critical Perspective on Action Research as a Method for Information Systems Research
Baskerville, R. L., & Wood-Harper, A. T. (1996) · Journal of Information Technology · doi:10.1177/026839629601100305 · Cited by 390 · cited 2×
“Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study and field study as observational methods (Zelkowitz and Wallace 1998).”
BackgroundA Model for Technology Transfer in Practice
Gorschek, T., Garre, P., Larsson, S., & Wohlin, C. (2006) · IEEE Software · doi:10.1109/ms.2006.147 · Cited by 350 · cited 2×
“2004) and technology transfer studies (Gorschek et al. 2006), the research method should be characterized as action research.”
BackgroundMethodExperiences from Conducting Semi-structured Interviews in Empirical Software Engineering Research
Hove, S. E., & Anda, B. (2005) · 11th IEEE International Software Metrics Symposium (METRICS'05) · doi:10.1109/metrics.2005.24 · Cited by 345 · cited 2×
“Possibly a dedicated and trained scribe may capture sufficient detail in real-time, but the recording should at least be done as a backup (Hove and Anda 2005).”
BackgroundControlling action research projects
Avison, D., Baskerville, R., & Myers, M. (2001) · Information Technology & People · doi:10.1108/09593840110384762 · Cited by 326 · cited 2×
“Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study and field study as observational methods (Zelkowitz and Wallace 1998).”
BackgroundReporting guidelines for controlled experiments in software engineering
Jedlitschka, A., & Pfahl, D. (2005) · 2005 International Symposium on Empirical Software Engineering, 2005. · doi:10.1109/isese.2005.1541818 · Cited by 295 · cited 1×
“…2) its conclusions are based on a clear chain of evidence, whether qualitative or quantitative, collected from multiple sources in a planned and consistent manner, and 3) it adds to existing knowledge by being based on previously established theory, if such exist, or by building theory.”
BackgroundEmpirical Research Methods in Software Engineering
Wohlin, C., Höst, M., & Henningsson, K. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-45143-3_2 · Cited by 270
Self-citationResearch Design for Social Work and the Human Services
Anastas, J. W. (2000) · Columbia University Press eBooks · doi:10.7312/anas11890 · Cited by 251 · cited 1×
“The research process may be characterized as fixed or flexible according to Anastas and MacDonald (1994) and Robson (2002).”
MethodBuilding Theories in Software Engineering
Sjøberg, D. I. K., Dybå, T., Anda, B. C. D., & Hannay, J. E. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_12 · Cited by 244 · cited 1×
“However, Sjøberg et al., describe a framework for theories including constructs of interest, relations between constructs, explanations to the relations, and scope of the theory (Sjøberg et al. 2008).”
BackgroundManaging risk in software process improvement: an action research approach
Iversen, J. H., Mathiassen, L., & Nielsen, P. A. (2004) · MIS Quarterly · Cited by 239
A Systematic Review of Theory Use in Software Engineering Experiments
Hannay, J. E., Sjoberg, D. I. K., & Dyba, T. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.12 · Cited by 226 · cited 2×
“Communicating research results in terms of theories is an underdeveloped practice in software engineering (Hannay et al. 2007).”
BackgroundRegnell, B., Höst, M., och Dag, J. N., Beremark, P., & Hjelm, T. (2001) · Requirements Engineering · doi:10.1007/s007660170015 · Cited by 206 · cited 2×
“More information about the studies can be found in the original publications (Karlström and Runeson 2005; 2006) (XP), (Regnell et al. 2001)”
BackgroundSelf-citationEthical issues in empirical studies of software engineering
Singer, J., & Vinson, N. G. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1158289 · Cited by 193 · cited 2×
“Data collected from and opinions stated by individual employees may be sensitive if presented e.g. to their managers (Singer and Vinson 2002).”
BackgroundCombining Agile Methods with Stage-Gate Project Management
Karlstrom, D., & Runeson, P. (2005) · IEEE Software · doi:10.1109/ms.2005.59 · Cited by 187 · cited 3×
“More information about the studies can be found in the original publications (Karlström and Runeson 2005; 2006) (XP), (Regnell et al. 2001)”
BackgroundMethodSelf-citationHints for Reviewing Empirical Work in Software Engineering
Tichy, W. F. (2000) · Empirical Software Engineering · doi:10.1023/a:1009844119158 · Cited by 176 · cited 1×
“Reviewing empirical research in general must be done with certain care (Tichy 2000).”
BackgroundManaging Risk in Software Process Improvement: An Action Research Approach1
Iversen, J. H., Mathiassen, L., & Nielsen, P. A. (2004) · MIS Quarterly · doi:10.2307/25148645 · Cited by 173 · cited 2×
“In software process improvement (Dittrich et al. 2008; Iversen et al. 2004) and technology transfer studies (Gorschek et al.”
MethodAn Ethnographic Study of XP Practice
Sharp, H., & Robinson, H. (2004) · Empirical Software Engineering · doi:10.1023/b:emse.0000039884.79385.54 · Cited by 167
Integrating agile software development into stage-gate managed product development
Karlström, D., & Runeson, P. (2006) · Empirical Software Engineering · doi:10.1007/s10664-006-6402-8 · Cited by 156 · cited 9×
“Study XP was, for example, presented to the involved companies in seminar format, and to the research community in journal format (Karlström and Runeson 2006) , to practitioners in a magazine format (Karlström and Runeson 2005) , and in the form of a Ph.D. thesis (Karlström 2004).”
BackgroundMethodInfluentialSelf-citationEthnographically-informed empirical studies of software practice
Robinson, H., Segal, J., & Sharp, H. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.007 · Cited by 151 · cited 2×
“Further, it is particularly relevant to use observations, where it is suspected that there is a deviation between an “official” view of matters and the “real” case (Robinson et al. 2007).”
BackgroundChecklists for Software Engineering Case Study Research
Host, M., & Runeson, P. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.46 · Cited by 145
Self-citationReplicating software engineering experiments: addressing the tacit knowledge problem
Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonca, M., & Fabbri, S. (2003) · Proceedings International Symposium on Empirical Software Engineering · doi:10.1109/isese.2002.1166920 · Cited by 142 · cited 2×
“…2) its conclusions are based on a clear chain of evidence, whether qualitative or quantitative, collected from multiple sources in a planned and consistent manner, and 3) it adds to existing knowledge by being based on previously established theory, if such exist, or by building theory.”
BackgroundEvaluating guidelines for reporting empirical software engineering studies
Kitchenham, B. A., Al-Kilidar, H., Babar, M. A., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., & Zhu, L. (2007) · Empirical Software Engineering · doi:10.1007/s10664-007-9053-5 · Cited by 138
A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems
Grbac, T. G., Runeson, P., & Huljenić, D. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.46 · Cited by 138
Self-citationA Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems
Andersson, C., & Runeson, P. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.1005 · Cited by 135 · cited 6×
“Hypothesis generation may take place within one cycle of a case study, or with data from one unit of analysis, and hypothesis confirmation may be done with data from another cycle or unit of analysis (Andersson and Runeson 2007b).”
BackgroundMethodResultInfluentialSelf-citationIdentification of dynamic comprehension processes during large scale maintenance
Von Mayrhauser, A., & Vans, A. M. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.508315 · Cited by 124
Dittrich, Y., Rönkkö, K., Eriksson, J., Hansson, C., & Lindeberg, O. (2007) · Empirical Software Engineering · doi:10.1007/s10664-007-9057-1 · Cited by 122 · cited 2×
“In software process improvement (Dittrich et al. 2008; Iversen et al. 2004) and technology transfer studies (Gorschek et al.”
MethodThe Goal/Question/Metric Method:
van Solingen, R., & Berghout, E. W. (2000) · Cited by 114
Research in computer science: an empirical study
Ramesh, V., Glass, R. L., & Vessey, I. (2003) · Journal of Systems and Software · doi:10.1016/s0164-1212(03)00015-3 · Cited by 113 · cited 1×
“However, the overall share of empirical studies is negligibly small in computer science research; Sjøberg et al. (2005), found 103 experiments in 5,453 articles Ramesh et al. (2004) and identified less than 2% experiments with human subjects, and only 0.16% field studies among 628 articles.”
BackgroundInfluentialFor the Special issue on Qualitative Software Engineering Research
Dittrich, Y., John, M., Singer, J., & Tessem, B. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.009 · Cited by 97
Europump extends quarterly order intake surveys
Robson, G. (2002) · World Pumps · doi:10.1016/s0262-1762(02)80276-0 · Cited by 90
Improving Software Development Management through Software Project Telemetry
Johnson, P. M., Kou, H., Paulding, M., Zhang, Q., Kagawa, A., & Yamashita, T. (2005) · IEEE Software · doi:10.1109/ms.2005.95 · Cited by 84 · cited 1×
“This approach is, for example taken in Software Project Telemetry (Johnson et al. 2005) where the usage of software engineering tools is automatically monitored, and observed through video recording.”
MethodDesigning a case study protocol for application in IS research
Pervan, G., & Maimbo, M. (2005) · eSpace (Curtin University) · Cited by 64
Fault-Prone Filtering: Detection of Fault-Prone Modules Using Spam Filtering Technique
Mizuno, O., Ikami, S., Nakaichi, S., & Kikuno, T. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.29 · Cited by 59
Can Programmers Reuse Software?
Woodfield, S. N., Embley, D. W., & Scott, D. T. (1987) · IEEE Software · doi:10.1109/ms.1987.231064 · Cited by 52
A spiral process model for case studies on software quality monitoring—method and metrics
Andersson, C., & Runeson, P. (2006) · Software Process Improvement and Practice · doi:10.1002/spip.311 · Cited by 51 · cited 6×
“Hypothesis generation may take place within one cycle of a case study, or with data from one unit of analysis, and hypothesis confirmation may be done with data from another cycle or unit of analysis (Andersson and Runeson 2007b).”
BackgroundMethodResultInfluentialSelf-citationHuman Centric Computing Languages and Environments
Hosking, J., & Cox, P. T. (2003) · Cited by 50
Collecting Feedback during Software Engineering Experiments
Karahasanovi�, A., Anda, B., Arisholm, E., Hove, S. E., J�rgensen, M., Sjberg, D., & Welland, R. (2005) · Empirical Software Engineering · doi:10.1007/s10664-004-6189-4 · Cited by 46 · cited 1×
“An alternative approach is presented by Karahasanovi ć et al. (2005) where a tool for sampling is used to obtain data and feedback from the participants.”
MethodBeg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research
Sim, S. E., Singer, J., & Storey, M. (2001) · Empirical Software Engineering · doi:10.1023/a:1009809824225 · Cited by 40
Protocol analysis: a neglected practice
Owen, S., Pearl Brereton, A., & Budgen, D. (2006) · Communications of the ACM · doi:10.1145/1113034.1113039 · Cited by 38 · cited 1×
“One approach is to monitor a group of software engineers with a video recorder and later on analyze the recording, for example through protocol analysis (Owen et al. 2006; von Mayrhauser and Vans 1996).”
BackgroundEthical Issues in Empirical Software Engineering: The Limits of Policy
Andrews, A. A., & Pradhan, A. S. (2001) · Empirical Software Engineering · doi:10.1023/a:1011442319273 · Cited by 34 · cited 3×
“In addition, this must take place under the balance between researcher ’ s duty and goal to publish their results, and the companies ’ and individuals ’ integrity (Amschler Andrews and Pradhan 2001).”
BackgroundInfluentialCase Studies for Software Engineers
Perry, D. E., Sim, S. E., & Easterbrook, S. (2005) · IEEE Software Engineering Workshop (SEW) · doi:10.1109/sew.2005.2 · Cited by 32
Methods for improving controlled experimentation in software engineering
Moher, T. G., & Schneider, G. (1981) · doi:10.5555/800078.802535 · Cited by 23
Building Theories from Multiple Evidence Sources
Shull, F. J., & Feldmann, R. L. (2007) · Springer eBooks · doi:10.1007/978-1-84800-044-5_13 · Cited by 22 · cited 1×
“Using theories to develop the research direction is not well established in the software engineering field, as concluded in a systematic review on the topic (Hannay et al. 2007; Shull and Feldman 2008).”
BackgroundAssertions in end-user software engineering: a think-aloud study
Wallace, C., Cook, C., Summet, J. W., & Burnett, M. (2003) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/hcc.2002.1046348 · Cited by 21 · cited 1×
“This can be combined with recording of audio and keystrokes as proposed e.g. by Wallace et al. (2002).”
BackgroundSpecial section: Controlled Experiments in Software Engineering
Wohlin, C., & Höst, M. (2001) · Information and Software Technology · doi:10.1016/s0950-5849(01)00200-2 · Cited by 10 · cited 1×
“For more information about quantitative analysis, refer for example to (Wohlin et al. 2000; Wohlin and Höst 2001; Kitchenham et al. 2002).”
MethodSelf-citationCombining qualitative empirical research with method, technique and process improvement
Dittrich, Y., Rönkkö, K., Eriksson, J., Hansson, C., & Lindeberg, O. (2008) · Cited by 7
Integrating Management and Engineering Processes in Software Product Development
Karlström, D. (2004) · Lund University Publications (Lund University) · Cited by 4
Research works printed but not linked: 2
- Shull F, Singer J, Sjøberg D (eds) (2008) Guide to Advanced Empirical Software Engineering. Springer- Verlag: London
- Martin Höst is an Associate Professor in Software Engineering in the Software Engineering Research Group at the Department of Computer Science, Lund University, Sweden. He received an M.Sc. degree from Lund University in 1992 and a Ph.D. degree in Software Engineering from the same university in 1999. His main research interests include software process improvement, software quality, empirical software engineering, and computer simulation of software development processes. The research is mainly conducted through empirical methods such as case studies, controlled experiments, and surveys. He has published more than 40 papers in international journals and conference proceedings.
Cited by
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
“ESEM P41 S36 (a,b) Kitchenham, B.A., Brereton, P., Turner, M., Niazi, M.K., Linkman, S., Pretorius, R., Budgen, D. 2010 Refining the systematic literature review process-two participantobserver case studies Empirical Software Engineering 15, 618–653 P42 S37 Kitchenham, B.A., Andrew J. Burn, Zhi Li 2009 A Quality Check…”
BackgroundInfluentialDigital twins in smart farming
Verdouw, C., Tekinerdogan, B., Beulens, A., & Wolfert, S. (2021) · Agricultural Systems · doi:10.1016/j.agsy.2020.103046 · Cited by 711
Deep learning code fragments for code clone detection
White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558
“Judging code clones is inherently difficult (even among experts [102,104,107]) because of imperfect definitions [10,106] and the lack of oracles [107], so we developed a research instrument [109] to support consistent evaluations and control construct threats.”
BackgroundMatcha, W., Uzir, N. A., Gasevic, D., & Pardo, A. (2020) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2019.2916802 · Cited by 399
“We particularly refer to some technology-related fields such as empirical software engineering from which empirical research on user-centered learning analytics systems can learn and which have guidelines how results should be reported and threats to validity discussed [99], [100].”
BackgroundEducational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., et al. (2015) · Proceedings of the 2015 ITiCSE on Working Group Reports · doi:10.1145/2858796.2858798 · Cited by 332
“For this purpose, we adapted the quality criteria for research proposed by Downs & Black [34] and Runeson & H¨ost [94] and added them to the data extraction form.”
MethodInfluential
Metadata
- Authors
- Runeson, P., & Höst, M.
- Affiliations
- Lund University (SE)
- Corresponding author
- Runeson, Per
- Venue
- Empirical Software Engineering
- Published
- 2008-12-19
- Volume
- 14
- Issue
- 2
- Pages
- 131-164
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W2162739315
- Cited by
- 3,862
- References
- 75
- Citations by year
- 2026: 204 · 2025: 225 · 2024: 244 · 2023: 241 · 2022: 246 · 2021: 299 · 2020: 312 · 2019: 268 · 2018: 280 · 2017: 237 · 2016: 260 · 2015: 263 · 2014: 252 · 2013: 198 · 2012: 151
- FWCI
- 186.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Empirical Software Engineering (publishedVersion, cc-by-nc)
- Lund University Publications (Lund University) (submittedVersion, other-oa)
- Funders
- Vetenskapsrådet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Sociology and Political Science › Qualitative Comparative Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- case study method, software engineering research, reporting guidelines, research quality, naturalistic study, qualitative research practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.





