The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Sampling in software engineering research: a critical review and guidelines
Open accessAbstract
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
(1984) · Cited by 81,983
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,535
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Qualitative Data Analysis: A Methods Sourcebook
Miles, M. B., Huberman, A. M., & Saldaña, J. (1988) · Cited by 20,701
(2014) · Cited by 15,747
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Qualitative Data Analysis: A Sourcebook of New Methods
Lotto, L. S., Miles, M. B., & Huberman, A. M. (1986) · Educational Evaluation and Policy Analysis · doi:10.2307/1163741 · Cited by 10,315
(2014) · Medical Entomology and Zoology · Cited by 6,828
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1954) · Journal of Experimental Psychology · doi:10.1037/h0055392 · Cited by 6,466 · cited 1×
“Most general methods textbooks, such a Trochim’s excellent Research Methods Knowledge Base (Trochim and Donnelly 2001) and SE-specific methods texts (e.g. Wohlin et al. 2012; Foster 2014; Easterbrook et al. 2008) discuss the logic of sampling, and many of the strategies described in Section 2.”
The information capacity of the human motor system in controlling the amplitude of movement.
Fitts, P. M. (1992) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.121.3.262 · Cited by 6,278
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2008) · Cited by 5,345
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
Internet, Phone, Mail, and Mixed‐Mode Surveys: The Tailored Design Method
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014) · Wiley eBooks · doi:10.1002/9781394260645 · Cited by 4,460
Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations
Heckathorn, D. D. (1997) · Social Problems · doi:10.1525/sp.1997.44.2.03x0221m · Cited by 4,409
The neglected 95%: Why American psychology needs to become less American.
Arnett, J. J. (2008) · American Psychologist · doi:10.1037/0003-066x.63.7.602 · Cited by 2,019
Statistical Rethinking: A Bayesian Course with Examples in R and Stan
McElreath, R. (2020) · doi:10.1201/9780429029608 · Cited by 1,959
Phenomenology of Practice: Meaning-Giving Methods in Phenomenological Research and Writing
van Manen, M. (2014) · Cited by 1,556
Generalizing Generalizability in Information Systems Research
Lee, A. S., & Baskerville, R. L. (2003) · Information Systems Research · doi:10.1287/isre.14.3.221.16560 · Cited by 1,551
Research methods knowledge base
Trochim, W. M. K., Donnelly, J. P., & Donnelly, J. P. (2001) · Cited by 1,493
Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys
Sax, L. J., Gilmartin, S. K., & Bryant, A. N. (2003) · Research in Higher Education · doi:10.1023/a:1024232915870 · Cited by 1,294
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
Epistemological and methodological bases of naturalistic inquiry
Guba, E. G., & Lincoln, Y. S. (1982) · ECTJ · doi:10.1007/bf02765185 · Cited by 1,186
Selecting Empirical Methods for Software Engineering Research
Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_11 · Cited by 1,167 · cited 2×
“Most general methods textbooks, such a Trochim’s excellent Research Methods Knowledge Base (Trochim and Donnelly 2001) and SE-specific methods texts (e.g. Wohlin et al. 2012; Foster 2014; Easterbrook et al. 2008) discuss the logic of sampling, and many of the strategies described in Section 2.”
MethodSampling hard to reach populations
Faugier, J., & Sargeant, M. (1997) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.1997.00371.x · Cited by 1,050
Epistemological and Methodological Bases of Naturalistic Inquiry
Guba, E. G., & Lincoln, Y. S. (1983) · Evaluation Models · doi:10.1007/978-94-009-6669-7_18 · Cited by 979
Sampling in Qualitative Research: Insights from an Overview of the Methods Literature
Gentles, S., Charles, C., Ploeg, J., & McKibbon, K. A. (2015) · The Qualitative Report · doi:10.46743/2160-3715/2015.2373 · Cited by 771
Action Research: Its Nature and Validity
Checkland, P., & Holwell, S. (1998) · Systemic Practice and Action Research · doi:10.1023/a:1022908820784 · Cited by 731
Sampling Essentials: Practical Guidelines for Making Sampling Choices
Daniel, J. (2012) · Sage eBooks · doi:10.4135/9781452272047 · Cited by 668
Local Standards for Sample Size at CHI
Caine, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858498 · Cited by 657
Theoretical Sampling and Category Development in Grounded Theory
Draucker, C. B., Martsolf, D. S., Ross, R., & Rusk, T. B. (2007) · Qualitative Health Research · doi:10.1177/1049732307308450 · Cited by 600 · cited 3×
“In the grounded theory literature, theoretical sampling refers to selecting items to study based on an emerging theory (e.g. Breckenridge and Jones 2009; Draucker et al. 2007).”
BackgroundPhenomenology of Practice: Meaning-Giving Methods in Phenomenological Research and Writing
van Manen, M. (2016) · Routledge eBooks · doi:10.4324/9781315422657 · Cited by 567
Thompson, S. K. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474975 · Cited by 537
Motivation in software engineering: A systematic literature review
Beecham, S., Baddoo, N., Hall, T., Robinson, H., & Sharp, H. (2008) · Information and Software Technology · doi:10.1016/j.infsof.2007.09.004 · Cited by 506
Kitchenham, B. A., & Pfleeger, S. L. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_3 · Cited by 484
Grounded theory in software engineering research: a critical review and guidelines
Stol, K.-J., Ralph, P., & Fitzgerald, B. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884833 · Cited by 470 · cited 5×
“Critical reviews in software engineering often investigate methodological topics; for example, how grounded theory and ethnography are reported (Stol et al. 2016; Zhang et al. 2019) or how qualitative research is synthesized (Huang et al. 2018).”
BackgroundMethodInfluentialSelf-citationMalekinejad, M., Johnston, L. G., Kendall, C., Kerr, L. R. F. S., Rifkin, M. R., & Rutherford, G. W. (2008) · AIDS and Behavior · doi:10.1007/s10461-008-9421-1 · Cited by 433 · cited 2×
“If respondent-driven sampling (Section 2.1.4) can reach injecting-drug users and sex workers (Malekinejad et al. 2008), surely it can help reach software developers.”
InfluentialThe Qualitas Corpus: A Curated Collection of Java Code for Empirical Studies
Tempero, E., Anslow, C., Dietrich, J., Han, T., Li, J., Lumpe, M., Melton, H., & Noble, J. (2010) · 2010 Asia Pacific Software Engineering Conference · doi:10.1109/apsec.2010.46 · Cited by 407
The GHTorent dataset and tool suite
Gousios, G. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624034 · Cited by 399
Assessing respondent-driven sampling
Goel, S., & Salganik, M. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1000261107 · Cited by 385
Henry, G. (1990) · Sage eBooks · doi:10.4135/9781412985451 · Cited by 376
Gender and Tenure Diversity in GitHub Teams
Vasilescu, B., Posnett, D., Ray, B., van den Brand, M. G. J., Serebrenik, A., Devanbu, P., & Filkov, V. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702549 · Cited by 354
Sampling Hard-to-Reach Populations with Respondent Driven Sampling
Johnston, L. G., & Sabin, K. (2010) · Methodological Innovations Online · doi:10.4256/mio.2010.0017 · Cited by 331
Principles of survey research: part 5: populations and samples
Kitchenham, B., & Pfleeger, S. L. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/571681.571686 · Cited by 329
An empirical study of open-source and closed-source software products
Paulson, J. W., Succi, G., & Eberlein, A. (2004) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2004.1274044 · Cited by 312
Statistically nonrepresentative stratified sampling: A sampling technique for qualitative studies
Trost, J. E. (1986) · Qualitative Sociology · doi:10.1007/bf00988249 · Cited by 309
Taxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380395 · Cited by 291
Demystifying Theoretical Sampling
Breckenridge, J. P., & Jones, D. (2009) · The grounded theory review · Cited by 273
Pandemic programming: How COVID-19 affects software developers and how their organizations can help
Ralph, P., Baltes, S., Adisaputri, G., Torkar, R., Kovalenko, V., Kalinowski, M., Novielli, N., Yoo, S., Devroey, X., Tan, X., et al. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09875-y · Cited by 269
Self-citationEmpirical software engineering experts on the use of students and professionals in experiments
Falessi, D., Juristo, N., Wohlin, C., Turhan, B., Münch, J., Jedlitschka, A., & Oivo, M. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9523-3 · Cited by 257
The ABC of Software Engineering Research
Stol, K.-J., & Fitzgerald, B. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3241743 · Cited by 245
Conducting realistic experiments in software engineering
Sjoberg, D. I. K., Anda, B., Arisholm, E., Dyba, T., Jorgensen, M., Karahasanovic, A., Koren, E. F., & Vokac, M. (2003) · Proceedings International Symposium on Empirical Software Engineering · doi:10.1109/isese.2002.1166921 · Cited by 197
Representative Sampling, III: The Current Statistical Literature
Kruskal, W., & Mosteller, F. (1979) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1402647 · Cited by 194
Patterns of Knowledge in API Reference Documentation
Maalej, W., & Robillard, M. P. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.12 · Cited by 183
Diversity in software engineering research
Nagappan, M., Zimmermann, T., & Bird, C. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491415 · Cited by 171
SOTorrent: reconstructing and analyzing the evolution of stack overflow posts
Baltes, S., Dumani, L., Treude, C., & Diehl, S. (2018) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3196398.3196430 · Cited by 167
Self-citationRepresentative Sampling, I: Non-Scientific Literature
Kruskal, W., & Mosteller, F. C. (1979) · International Statistical Review · doi:10.2307/1403202 · Cited by 131
On the success of empirical studies in the international conference on software engineering
Zannier, C., Melnik, G., & Maurer, F. (2006) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1134285.1134333 · Cited by 124
Toward Methodological Guidelines for Process Theories and Taxonomies in Software Engineering
Ralph, P. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2796554 · Cited by 118
Self-citationSedano, T., Ralph, P., & Péraire, C. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00036 · Cited by 114
Self-citationSOTorrent: Reconstructing and Analyzing the Evolution of Stack Overflow Posts
Baltes, S., Dumani, L., Treude, C., & Diehl, S. (2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · doi:10.48550/arxiv.1803.07311 · Cited by 107
Self-citationEmpirical Standards for Software Engineering ResearchPreprint
Ralph, P., bin Ali, N., Baltes, S., Bianculli, D., Díaz, J., Dittrich, Y., Ernst, N., Felderer, M., FELDT, R. H., Filieri, A., et al. (2020) · arXiv (Cornell University) · doi:10.4230/lipics.esem.2026.10 · Cited by 96
Self-citationCosentino, V., Luis, J., & Cabot, J. (2016) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2901739.2901776 · Cited by 89
Hoeven, L. V., Janssen, M. P., Roes, K. C. B., & Koffijberg, H. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0089-8 · Cited by 88
Gender Differences in Personality Traits of Software Engineers
Russo, D., & Stol, K.-J. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3003413 · Cited by 74
FELDT, R. H., Zimmermann, T., Bergersen, G. R., Falessi, D., Jedlitschka, A., Juristo, N., Münch, J., Oivo, M., Runeson, P., Shepperd, M., et al. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9655-0 · Cited by 66
Synthesizing qualitative research in software engineering: a critical review
Huang, X., Zhang, H., Zhou, X., Babar, M. A., & Yang, S. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180235 · Cited by 57
Worse Than Spam: Issues In Sampling Software Developers
Baltes, S., & Diehl, S. (2016) · Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962628 · Cited by 54
Self-citationInvestigating probabilistic sampling approaches for large-scale surveys in software engineering
de Mello, R., da Silva, P. C., & Travassos, G. H. (2015) · Journal of Software Engineering Research and Development · doi:10.1186/s40411-015-0023-0 · Cited by 50
Effects of Incentives in Surveys
Toepoel, V. (2012) · Springer eBooks · doi:10.1007/978-1-4614-3876-2_13 · Cited by 37
Surveys in Software Engineering: Identifying Representative Samples
de Mello, R., & Travassos, G. H. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962632 · Cited by 34
Ethnographic research in software engineering: a critical review and checklist
Zhang, H., Huang, X., Zhou, X., Huang, H., & Babar, M. A. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338976 · Cited by 28
Worse Than Spam: Issues In Sampling Software DevelopersPreprint
Baltes, S., & Diehl, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00838 · Cited by 26
Self-citationThere is no random sampling in software engineering research
Amir, B., & Ralph, P. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3195001 · Cited by 26
Self-citationBiomedical Research Methodology
Das, P. (2011) · Jaypee Brothers Medical Publishers (P) Ltd. eBooks · doi:10.5005/jp/books/11438 · Cited by 19
Salleh, N., Hoda, R., Su, M. T., Kanij, T., & Grundy, J. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.12.001 · Cited by 18
Software Engineering Research at the International Conference on Software Engineering in 2016
Theisen, C., Dunaiski, M., Williams, L. A., & Visser, W. (2018) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3149485.3149496 · Cited by 12
How software practitioners use informal local meetups to share software engineering knowledge
Ingram, C., & Drachen, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380333 · Cited by 12
Cited by
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
“” This time efficiency can be attributed to the NLP capabilities of LLMs, which enable users to communicate their needs more effectively and rapidly obtain tailored solutions [14].”
BackgroundWyrich, M., Preikschat, A., Graziotin, D., & Wagner, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00055 · Cited by 24
Practitioners’ expectations on log anomaly detection
Ma, X., Li, Y., Keung, J., Yu, X., Zou, H., Yang, Z., Sarro, F., & Barr, E. T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586700 · Cited by 21
“For interviews, we employ expert sampling, a subtype of purposive sampling [14], to target professionals with relevant expertise in log analysis to gather valuable insights.”
Accountability in Code Review: The Role of Intrinsic Drivers and the Impact of LLMs
Alami, A., Jensen, V., & Ernst, N. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3721127 · Cited by 11
On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review
Baltes, S., Speith, T., Chiteri, B., Mohsenimofidi, S., Chakraborty, S., & Buschek, D. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3659804 · Cited by 5
Self-citation
Metadata
- Authors
- Baltes, S., & Ralph, P.
- Venue
- Empirical Software Engineering
- Published
- 2022-04-28
- Volume
- 27
- Issue
- 4
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W3006568433
- Cited by
- 382
- References
- 81
- FWCI
- 73.69
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.