The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Qualitative methods in empirical studies of software engineering
Abstract
While empirical studies in software engineering are beginning to gain recognition in the research community, this subarea is also entering a new level of maturity by beginning to address the human aspects of software development. This added focus has added a new layer of complexity to an already challenging area of research. Along with new research questions, new research methods are needed to study nontechnical aspects of software engineering. In many other disciplines, qualitative research methods have been developed and are commonly used to handle the complexity of issues involving human behaviour. The paper presents several qualitative methods for data collection and analysis and describes them in terms of how they might be incorporated into empirical studies of software engineering, in particular how they might be combined with quantitative methods. To illustrate this use of qualitative methods, examples from real software engineering studies are used throughout.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Building Theories from Case Study Research
Eisenhardt, K. M. (1989) · The Academy of Management Review · doi:10.2307/258557 · Cited by 17,455
People, organizations, and process improvement
Perry, D. E., Staudenmayer, N. A., & Votta, L. G. (1994) · IEEE Software · doi:10.1109/52.300082 · Cited by 401
Research works printed but not linked: 19
- curtis, 1986, by the way, did anyone study any real programmers?, Empirical Studies of Programmers, 256
- miles, 1994, Qualitative Data Analysis An Expanded Sourcebook
- parra, 1997, the package-based development process in the flight dynamics division, Proc 22nd Software Eng Workshop, 21
- 10.1109/52.526836 · DOI
- gilgun, 1992, definitions, methodologies, and methods in qualitative family research, Qualitative Methods in Family Research
- 10.1109/32.708569 · DOI
- 10.2307/2393551 · DOI · Paper page
- hackos, 1998, User and Task Analysis for Interface Design, 258
- judd, 1991, Research Methods in Social Relations, 298
- 10.1109/52.676747 · DOI
- 10.1145/800027.808439 · DOI
- lincoln, 1985, Naturalistic Inquiry
- 10.1109/32.508315 · DOI
- basili, 1986, experimentation in software engineering, IEEE Trans Software Engineering, 12, 733, 10.1109/TSE.1986.6312975 · DOI · Paper page
- mcconnell, 1996, Rapid Development Taming Wild Software Schedules, 249
- shneiderman, 1998, Designing the User Interface Strategies for Effective Human?Computer Interaction, 146
- glaser, 1967, The Discovery of Grounded Theory Strategies for Qualitative Research
- taylor, 1984, Introduction to Qualitative Research Methods
- whyte, 1984, Learning from the Field A Guide from Experience
Cited by
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862
Grounded theory in software engineering research: a critical review and guidelines
Stol, K.-J., Ralph, P., & Fitzgerald, B. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884833 · Cited by 470
Applying Systematic Reviews to Diverse Study Types: An Experience Report
Dyba, T., Dingsoyr, T., & Hanssen, G. K. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.59 · Cited by 391
Measure it? Manage it? Ignore it? software practitioners and technical debt
Ernst, N. A., Bellomo, S., Ozkaya, I., Nord, R. L., & Gorton, I. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786848 · Cited by 260
The ABC of Software Engineering Research
Stol, K.-J., & Fitzgerald, B. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3241743 · Cited by 245
Using grounded theory to study the experience of software development
Adolph, S., Hall, W., & Kruchten, P. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9152-6 · Cited by 235
Coleman, G., & O’Connor, R. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.011 · Cited by 204
Socio-Technical Grounded Theory for Software Engineering
Hoda, R. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3106280 · Cited by 204
Conducting on-line surveys in software engineering
Punter, T., Ciolkowski, M., Freimut, B., & John, I. (2003) · 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings. · doi:10.1109/isese.2003.1237967 · Cited by 193
Open source barriers to entry, revisited: a sociotechnical perspective
Mendez, C., Padala, H. S., Steine-Hanson, Z., Hilderbrand, C., Horvath, A., Hill, C., Simpson, L., Patil, N., Sarma, A., & Burnett, M. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180241 · Cited by 118
Studying the dialogue between users and developers of free apps in the Google Play Store
Hassan, S., Tantithamthavorn, C., Bezemer, C.-P., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9538-9 · Cited by 100
Ways of Knowing When Research Subjects Care
Howard, D., & Irani, L. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300327 · Cited by 70
Hidden Figures: Roles and Pathways of Successful OSS Contributors
Trinkenreich, B., Guizani, M., Wiese, I., Sarma, A., & Steinmacher, I. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415251 · Cited by 67
(2024) · doi:10.1145/3664646.3664773 · Cited by 67
Fairness-Aware Machine Learning Engineering: How Far Are We?
Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10402-y · Cited by 60
“To gather insights from practitioners, multiple empirical research methods could be exploited, e.g.,large-scalesurveyquestionnaires,structuredinterviews,literaturereviews,ordataaggre-gationofmajortrendsfromindustrialcasestudies(Seaman1999).”
BackgroundThe creative turn: new challenges for computing
Hugill, A., & Yang, H. (2013) · International Journal of Creative Computing · doi:10.1504/ijcrc.2013.056934 · Cited by 61
Mapping the Increasing Use of LLMs in Scientific Papers
Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.01268 · Cited by 55
How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow
Wang, S., Chen, T.-H., & Hassan, A. E. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2874470 · Cited by 50
Developers' code context models for change tasks
Fritz, T., Shepherd, D. C., Kevic, K., Snipes, W., & Bräunlich, C. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635905 · Cited by 45
Kuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44
Barcomb, A., Kaufmann, A., Riehle, D., Stol, K.-J., & Fitzgerald, B. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2872713 · Cited by 44
An Interview Study on Sustainability Concerns in Software Development Projects
Groher, I., & Weinreich, R. (2017) · 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2017.70 · Cited by 45
Older adults and hackathons: a qualitative study
Kopeć, W., Balcerzak, B., Nielek, R., Kowalik, G., Wierzbicki, A., & Casati, F. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9565-6 · Cited by 39
Foraging and navigations, fundamentally: developers' predictions of value and cost
Piorkowski, D., Henley, A. Z., Nabi, T., Fleming, S. D., Scaffidi, C., & Burnett, M. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950302 · Cited by 38
Using a grounded theory approach for exploring software product management challenges
Jantunen, S., & Gause, D. C. (2014) · Journal of Systems and Software · doi:10.1016/j.jss.2014.03.050 · Cited by 31
(2024) · doi:10.1145/3613904.3642939 · Cited by 31
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Empirical studies of requirements validation techniques
Raja, U. A. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909209 · Cited by 24
Gonçalves, M. K., González, V. M., & de Souza, L. R. B. (2011) · Verlag der Technischen Universität Graz · doi:10.3217/jucs-017-14-1913 · Cited by 22
An Empirical Study of Adoption of ChatGPT for Bug Fixing among Professional Developers
Ge, H., & Wu, Y. (2023) · Innovation & Technology Advances · doi:10.61187/ita.v1i1.19 · Cited by 20
Clear, T., Cajander, Å., Clear, A., McDermott, R., Daniels, M., Divitini, M., Forshaw, M., Humble, N., Kasinidou, M., Kleanthous, S., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709607 · Cited by 19
What are the characteristics of highly-selected packages? A case study on the npm ecosystem
Mujahid, S., Abdalkareem, R., & Shihab, E. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111588 · Cited by 16
Practitioners’ expectations on log anomaly detection
Ma, X., Li, Y., Keung, J., Yu, X., Zou, H., Yang, Z., Sarro, F., & Barr, E. T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586700 · Cited by 21
“We generate our interview questions from two sources: first principles and standard open-ended interviews [13].”
An empirical study on change-induced incidents of online service systems
Wu, Y., Chai, B., Li, Y., Liu, B., Li, J., Yang, Y., & Jiang, W. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00027 · Cited by 11
Security of Language Models for Code: A Systematic Literature Review
Chen, Y., Sun, W., Fang, C., Chen, Z., Ge, Y., Han, T., Zhang, Q., Liu, Y., Chen, Z., & Xu, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735554 · Cited by 10
Stray, V., Barbala, A., & Wivestad, V. T. (2025) · Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3696630.3730566 · Cited by 19
Qualitative study to identify icons characteristics on mobile phones applications interfaces
Ghayas, S., Sulaiman, S., Khan, M., & Jaafar, J. (2013) · 2013 IEEE Symposium on Wireless Technology & Applications (ISWTA) · doi:10.1109/iswta.2013.6688794 · Cited by 5
Metadata
- Authors
- Seaman, C. B.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 1999-07
- Volume
- 25
- Issue
- 4
- Pages
- 557-572
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2104577574
- Cited by
- 1,493
- References
- 2
- FWCI
- 16.16
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.