How does Machine Learning Change Software Development Practices?
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Adding an ability for a system to learn inherently adds uncertainty into the system. Given the rising popularity of incorporating machine learning into systems, we wondered how the addition alters software development practices. We performed a mixture of qualitative and quantitative studies with 14 interviewees and 342 survey respondents from 26 countries across four continents to elicit significant differences between the development of machine learning systems and the development of non-machine-learning systems. Our study uncovers significant differences in various aspects of software engineering (e.g., requirements, design, testing, and process) and work characteristics (e.g., skill variety, problem solving and task identity). Based on our findings, we highlight future research directions and provide recommendations for practitioners.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is complex for practitioners to comprehend the scopes and impacts of architectural changes [95, 103].”
From Software Architecture in Practice: Challenges and Opportunities (2023) · cites this paper as [103]
In the citing paper
“Moreover, software engineering has had decades of development in data management, which could be readily adapted to AI practices in many cases [190].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [190]
In the citing paper
“Exploratory processes in ML share many similarities to scientific programming, and would benefit from lessons learned in embracing uncertainty [190].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [190]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · cited 1×
“We conducted a thematic analysis [8] to process the recorded interviews by following the steps below: Transcribing and Coding.”
MethodA Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“We then use the Cohen’s Kappa measure [10] to examine the agreement between the two labelers.”
MethodHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808 · cited 1×
“[15], conducting 12 to 15 interviews of a homogeneous group is adequate to reach saturation.”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 2×
“Machine learning (ML) has progressed dramatically over the past three decades, from a laboratory curiosity to a practical technology in widespread commercial use [19].”
BackgroundHumphrey, S. E., Nahrgang, J. D., & Morgeson, F. P. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.5.1332 · Cited by 2,303 · cited 1×
“[18] in applied psychology, which consists of 21 work characteristics, e.”
BackgroundProbabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995 · cited 1×
“Uncertainty plays a fundamental role in probabilistic modeling [14]: Observed data can be consistent with various models, and thus which model is appropriate given the data is uncertain.”
BackgroundDeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
MethodSoftware Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 3×
“Most similar to our study is an empirical investigation of integrating AI capabilities into software and services and best practices from Microsoft teams [4].”
BackgroundMethodInfluential
Research works printed but not linked: 3
- romano, 2006, Appropriate statistics for ordinal level data: Should we really be using t-test and cohen'sd for evaluating group differences on the nsse and other surveys, Proc Annu Meet Florida Assoc Institutional Res, 1
- 10.1007/bf02729815 · doi:10.1007/bf02729815
- ng, 2017, Machine Learning Yearning
Cited by
Software Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
“com [21], AI model criteria relevant for end users [56], automatic version control in notebooks [98], or practices collected via practitioner surveys and/or interviews [203, 224].”
BackgroundThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Lo, S. K., Lu, Q., Wang, C., Paik, H.-Y., & Zhu, L. (2022) · ACM Computing Surveys · doi:10.1145/3450288 · Cited by 202
“olves a high number of participating clients’ collaboration which forms a large-scale distributed system. This calls for the software engineering thinking apart from the core machine learning concept [164]. Hence, the second motivation is to explore how to develop federated learning systems by examining existing studi…”
MethodNahar, N., Zhou, S., Lewis, G., & Kästner, C. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510209 · Cited by 131
“Reported problems relate to poor code quality (P1b, P2a, P3b, P5a, P6a-b, P10a, P11a, P14a, P15b-c, P17a, P18a, P19a, P20a-b, P26a) [9, 27, 34, 37, 74, 86, 105], insufficient documentation (P5a-b, P6a-b, P10a, P15c, P26a) [8, 46, 64, 113], and not extending version control to data and models (P3c, P7a, P10a, P14a, P20…”
BackgroundResultInfluentialGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
A survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 173
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
“Moreover, software engineering has had decades of development in data management, which could be readily adapted to AI practices in many cases [190].”
BackgroundHow AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151
A Survey on Data-driven Software Vulnerability Assessment and Prioritization
Le, T. H. M., Chen, H., & Babar, M. A. (2022) · ACM Computing Surveys · doi:10.1145/3529757 · Cited by 177
“Existing SV assessment and prioritization models for traditional systems have not considered unique data/model-related characteristics/features of data-driven systems [192].”
BackgroundGarg, S., Pundir, P., Rathee, G., Gupta, P. K., Garg, S., & Ahlawat, S. (2021) · 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE) · doi:10.1109/aike52691.2021.00010 · Cited by 91
What Makes Agile Software Development Agile?
Kuhrmann, M., Tell, P., Hebig, R., Klünder, J., Münch, J., Linssen, O., Pfahl, D., Felderer, M., Prause, C. R., MacDonell, S. G., et al. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3099532 · Cited by 132
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127
“While a wide body of academic literature acknowledges these challenges [4, 30, 32, 36, 56], there is little academic literature to guide practitioners.”
BackgroundArchitectural patterns for the design of federated learning systems
Lo, S. K., Lu, Q., Zhu, L., Paik, H.-Y., Xu, X., & Wang, C. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111357 · Cited by 149
Accessibility in Software Practice: A Practitioner’s Perspective
Bi, T., Xia, X., Lo, D., Grundy, J., Zimmermann, T., & Ford, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3503508 · Cited by 109
“Wan et al. performed a mixture of quality and quantitative studies with 14 interviews and 342 survey responses from 26 countries to elicit significant differences between the development of machine learning systems and the development of non-machine learning systems [48].”
BackgroundSelf-citationConstruction of a quality model for machine learning systems
Siebert, J., Jöckel, L., Heidrich, J., Trendowicz, A., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09557-y · Cited by 113
“…software components raises new challenges in comparison to “classical” software engineering (Arpteg et al., 2018; de Souza Nascimento et al., 2019; Kästner & Kang, 2020; Kumeno, 2020; Lwakatare et al., 2020; Sculley et al., 2015; Wan et al., 2019; Zhang & Tsai, 2002; Zhang et al., 2019).”
BackgroundThe art and practice of data science pipelines
Biswas, S., Wardat, M., & Rajan, H. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510057 · Cited by 128
“While designing a new DS pipeline [130], dividing roles in DS teams [65], defining software process in data-intensive setting [123], identifying best practices in AI and modularizing DS components [6], it is important to understand the current state of the DS pipeline, its variations and different…”
MethodChen, J., Xia, X., Lo, D., Grundy, J., & Yang, X. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10018-0 · Cited by 111
“Compared to the experiences of the respondents (including developers and researchers) of previous works, e.g., in machine learning (Wan et al. 2019) (min: 3, max: 16, median: 6, avg: 7.6 years), in desktop software development (Wan et al. 2018) (min: 3, max: 12, avg: 6.5 years), the smart contract…”
BackgroundInfluentialSelf-citationGonzalez, D., Zimmermann, T., & Nagappan, N. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387473 · Cited by 123
“Initial work has focused on interviews and surveys to understand how AI & ML projects are different [1, 54], and the challenges that developers face [3, 21, 37, 58].”
BackgroundA Literature Review of Using Machine Learning in Software Development Life Cycle Stages
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2021) · IEEE Access · doi:10.1109/access.2021.3119746 · Cited by 110
A Comparative Study of Class Rebalancing Methods for Security Bug Report Classification
Zheng, W., Xun, Y., Wu, X., Deng, Z., Chen, X., & Sui, Y. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3118026 · Cited by 85
Opportunities and Challenges in Data-Centric AI
Kumar, S., Datta, S., Singh, V., Singh, S. K., & Sharma, R. (2024) · IEEE Access · doi:10.1109/access.2024.3369417 · Cited by 111
“Therefore, an essential duty for an ML engineer to consider is assuring the data quality [7].”
BackgroundSoftware Architecture for ML-based Systems: What Exists and What Lies Ahead
Muccini, H., & Vaidhyanathan, K. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00026 · Cited by 87
Context-aware Retrieval-based Deep Commit Message Generation
Wang, H., Xia, X., Lo, D., He, Q., Wang, X., & Grundy, J. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3464689 · Cited by 70
“1%, which is similar to some previous studies in software engineering [11, 60, 61].”
ResultSelf-citationRequirements Engineering for Machine Learning: A Systematic Mapping Study
Villamizar, H., Escovedo, T., & Kalinowski, M. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00013 · Cited by 100
“A na ly se s P3 [29] Teaching Software Engineering for AI-Enabled Systems P4 [37] Emerging and changing tasks for developing ML systems P11 [53] Perspectives from data scientists P14 [21] Challenges and new directions of NFRs for ML P16 [33] How to adapt SQuaRE for ML-based AI systems P17 [24] How engineers perceive d…”
MethodNahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 86
Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and Challenges
Chen, H., & Babar, M. A. (2024) · ACM Computing Surveys · doi:10.1145/3638531 · Cited by 85
“To understand the difference and challenges, SWEBOK is introduced as the taxonomy for SE practices classification in [2, 82, 110, 164, 180].”
MethodSoftware Engineering by and for Humans in an AI Era
Abrahão, S., Grundy, J., Pezzè, M., Storey, M.-A., & Tamburri, D. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715111 · Cited by 37
“It also remains to be seen if the emergence of AI-powered tools that significantly change SE work and SE job expectations enables many who previously did not embark on an SE career to do so.”
Towards Guidelines for Assessing Qualities of Machine Learning Systems
Siebert, J., Joeckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-58793-2_2 · Cited by 58
“Developing and operating ML systems raises new challenges in comparison to “classical” software engineering [1, 2].”
BackgroundZheng, W., Gao, J., Wu, X., Liu, F., Xun, Y., Liu, G., & Chen, X. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110659 · Cited by 58
Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
Liang, J. T., Lin, M. M., Rao, N., & Myers, B. A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729342 · Cited by 75
Systematic literature review on software quality for AI-based software
Gezici, B., & Tarhan, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10105-2 · Cited by 69
“, randomness in data, initialization of weights, and optimizations) causes uncertainty and complicates software testing (Wan et al. 2019).”
BackgroundMethodInfluentialAdapting Software Architectures to Machine Learning Challenges
Serban, A., & Visser, J. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00029 · Cited by 71
Architecting ML-enabled systems: Challenges, best practices, and design decisions
Nazir, R., Bucaioni, A., & Pelliccione, P. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111860 · Cited by 57
Blomster, M., & Koivumäki, T. (2021) · Information Systems and e-Business Management · doi:10.1007/s10257-021-00547-y · Cited by 58
“In real life, projects that resemble software development are not isolated activities (Karlström and Runeson 2006; Wan et al. 2019).”
BackgroundMethodLorenzoni, G., Alencar, P. S. C., do Nascimento, N. M., & Cowan, D. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07574 · Cited by 49
“Most of the few existing approaches of SE to ML are focused on broadly ML workflows [1], [2], [4], [7], [9], [11]–[16], [20], [22]–[24].”
BackgroundWhat Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 62
“[82] found that knowing what groups of instances to create and how to group instances are still major challenges for many data scientists.”
BackgroundResultInfluentialHuman-Centered AI Design in Reality: A Study of Developer Companies’ Practices
Hartikainen, M., Väänänen, K., Lehtiö, A., Ala-Luopa, S., & Olsson, T. (2022) · Nordic Human-Computer Interaction Conference · doi:10.1145/3546155.3546677 · Cited by 48
Multilayered review of safety approaches for machine learning-based systems in the days of AI
Dey, S., & Lee, S.-W. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110941 · Cited by 53
What Do Users Ask in Open-Source AI Repositories? An Empirical Study of GitHub Issues
Yang, Z., Wang, C., Shi, J., Hoang, T., Kochhar, P. S., Lu, Q., Xing, Z., & Lo, D. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00024 · Cited by 63
Self-citationAn architecture for model-based and intelligent automation in DevOps
Eramo, R., Said, B., Oriol, M., Brunelière, H., & Morales, S. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112180 · Cited by 50
Rabe, M., Milz, S., & Mäder, P. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00023 · Cited by 44
“The development of machine learning applications shows substantial differences to traditional software development [65, 110, 112].”
BackgroundIdentifying algorithm in program code based on structural features using CNN classification model
Watanobe, Y., Rahman, M. M., Amin, M. F. I., & Kabir, R. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04078-y · Cited by 43
“ML capabilities can be integrated into a software system in many ways, including ML components, tools, libraries (cover ML functionalities), and frameworks [14].”
BackgroundInfluentialA reference architecture for the operationalization of machine learning models in manufacturing
Raffin, T., Reichenstein, T., Werner, J., Kühl, A., & Franke, J. E. (2022) · Procedia CIRP · doi:10.1016/j.procir.2022.10.062 · Cited by 43
Rao, N., Tsay, J., Kate, K., Hellendoorn, V. J., & Hirzel, M. J. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645203 · Cited by 48
“It uses a fine-tuned variant of the CodeT5 model [41] that we developed through experiments with a variety of neural models for program generation, ranging from training models from scratch to few-shot prompting large language models [25].”
MethodTowards a Human-Centred Artificial Intelligence Maturity Model
Hartikainen, M., Väänänen, K., & Olsson, T. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585752 · Cited by 40
“The use of data brings the requirement of AI systems to be fair and trustworthy [10, 12, 19, 25, 27].”
BackgroundSoftware engineering practices for machine learning — Adoption, effects, and team assessment
Serban, A., van der Blom, K., Hoos, H. H., & Visser, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111907 · Cited by 41
Towards Ethical Data-Driven Software: Filling the Gaps in Ethics Research & Practice
Johnson, B., & Smith, J. (2021) · Workshop on Ethics in Software Engineering Research and Practice · doi:10.1109/sethics52569.2021.00011 · Cited by 47
“Advances in software development, such as the integration of machine learning that uses real world data for decision making, have changed how we build software systems [5], [6] and forced developers to account for additional concerns [7].”
BackgroundFLRA: A Reference Architecture for Federated Learning Systems
Lo, S. K., Lu, Q., Paik, H.-Y., & Zhu, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86044-8_6 · Cited by 53
Requirements Engineering in Machine Learning Projects
Gjorgjevikj, A., Mishev, K., Antovski, L., & Trajanov, D. (2023) · IEEE Access · doi:10.1109/access.2023.3294840 · Cited by 38
“Wan et al. [21] present analyses of information obtained from 14 interviewees and 342 survey respondents from 26 countries.”
BackgroundWilhjelm, C., & Younis, A. A. (2020) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c51114.2020.00078 · Cited by 33
Developers’ Perceptions on the Impact of Chatgpt in Software Development: A SurveyPreprint
Vaillant, T. S., de Almeida, F. D., Silveira, P., Gao, C., Bosch, J., & de Almeida, E. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5023364 · Cited by 35
The AI Effect: Working at the Intersection of AI and SE
Carleton, A. D., Harper, E., Menzies, T., Xie, T., Eldh, S., & Lyu, M. R.-T. (2020) · IEEE Software · doi:10.1109/ms.2020.2987666 · Cited by 33
Software Architecture in Practice: Challenges and Opportunities
Wan, Z., Zhang, Y., Xia, X., Jiang, Y., & Lo, D. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616367 · Cited by 30
Self-citationTechnical Debt in the Peer-Review Documentation of R Packages: a rOpenSci Case Study
Codabux, Z., Vidoni, M. C., & Fard, F. H. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00032 · Cited by 36
From a Data Science Driven Process to a Continuous Delivery Process for Machine Learning Systems
Lwakatare, L. E., Crnkovic, I., Rånge, E., & Bosch, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-64148-1_12 · Cited by 28
Software Engineering Practice in the Development of Deep Learning ApplicationsPreprint
Zhang, X., Yang, Y., Feng, Y., & Chen, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03156 · Cited by 33
“development activities. Daka et al. [8] surveyed 225 developers to understand unit testing practices such as motivation of developers, their usage of automation tools, and their challenges. Wan et al.[31]performedamixtureofqualitativeandquantitativestudies with14interviewees and342questionnairerespondentstoinvestigate…”
InfluentialHeyn, H.-M., Knauss, E., & Pelliccione, P. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111604 · Cited by 34
An integration engineering framework for machine learning in healthcare
Assadi, A., Laussen, P. C., Goodwin, A. J., Goodfellow, S., Dixon, W., Greer, R. W., Jegatheeswaran, A., Singh, D., McCradden, M., Gallant, S. N., et al. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.932411 · Cited by 20
“Frontiers in Digital Health 04 development and deployment of Machine Learning models, they have been difficult and expensive to maintain and integrate (35).”
BackgroundInfluentialRevisiting the Identification of the Co-evolution of Production and Test Code
Sun, W., Yan, M. L., Liu, Z., Xia, X., Lei, Y., & Lo, D. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607183 · Cited by 25
“In previous empirical works [79, 80], when the manual analysis work is nearing its end, researchers claim that the collected cards/tags reach saturation when new cards/tags do not appear anymore.”
BackgroundResultInfluentialSelf-citationDevelopment in times of hype: How freelancers explore Generative AI?
Dolata, M., Lange, N., & Schwabe, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639111 · Cited by 35
“First, it outlines challenges identi fi ed by freelancers who create GenAI-based solutions, adding a new, freelance-oriented perspective to SE4AI literature [15, 16, 37, 38, 42, 45, 57].”
BackgroundTailoring Requirements Engineering for Responsible AI
Maalej, W., Pham, Y. D., & Chazette, L. (2023) · Computer · doi:10.1109/mc.2023.3243182 · Cited by 28
Interpretation-based Code Summarization
Geng, M., Wang, S., Dong, D., Wang, H., Cao, S., Zhang, K., & Jin, Z. G. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00026 · Cited by 29
“This work was supported in part by Excellent Youth Foundation of Hunan Province under Grant No. 2021JJ10050. code summarization) an active research domain in the software engineering community [10]–[12].”
BackgroundTesting Machine Learning Systems in Industry: An Empirical Study
Li, S., Guo, J., Lou, J., Fan, M. Y., Liu, T., & Zhang, D. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip55303.2022.9793981 · Cited by 27
Software quality prediction using machine learning
Alaswad, F., & Poovammal, E. (2022) · Materials Today Proceedings · doi:10.1016/j.matpr.2022.03.165 · Cited by 21
Practices for Managing Machine Learning Products: A Multivocal Literature Review
Alves, I., Leite, L., Meirelles, P., Kon, F., & Rocha, C. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3287759 · Cited by 28
Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach
Noei, S., Li, H., & Zou, Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705309 · Cited by 23
More sustainable artificial intelligence systems through stakeholder involvement?
Kunkel, S., Schmelzle, F., Niehoff, S., & Beier, G. (2023) · GAIA - Ecological Perspectives for Science and Society · doi:10.14512/gaia.32.s1.10 · Cited by 20
An extension of iStar for Machine Learning requirements by following the PRISE methodology
Barrera, J. M., Reina, A. R., Lavalle, A., Maté, A., & Trujillo, J. C. (2023) · Computer Standards & Interfaces · doi:10.1016/j.csi.2023.103806 · Cited by 22
Kotenko, I. V., Евгеньевич, И. К., Buinevich, M., Saenko, I. B., & Shorey, R. (2023) · Energies · doi:10.3390/en16135111 · Cited by 20
Clones in deep learning code: what, where, and why?
Jebnoun, H., Rahman, M. S., Khomh, F., & Muse, B. A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10099-x · Cited by 25
“Wan et al.[99] studied the features and impacts of machine learning on software development.”
BackgroundMaking Sense of AI Systems Development
Dolata, M., & Crowston, K. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3338857 · Cited by 22
“This adds to previous studies which were conducted in Asia [4] or North America [16], [23], the-orized based on literature review [9], [24], or attended to a single group of professionals like developers or data scientists [16], [23], [28].”
InfluentialHow far are we with automated machine learning? characterization and challenges of AutoML toolkits
Alamin, M. A. A., & Uddin, G. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10450-y · Cited by 22
Indykov, V., Strüber, D., & Wohlrab, R. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112373 · Cited by 22
Code Reviewer Recommendation in Tencent: Practice, Challenge, and Direction*
Chen, Q., Kong, D., Bao, L., Sun, C., Xia, X., & Li, S. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip55303.2022.9794124 · Cited by 22
Self-citationA Catalogue of Concerns for Specifying Machine Learning-Enabled Systems
Villamizar, H., Kalinowski, M., & Lopes, H. (2022) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298262.25-12 · Cited by 17
“Literature has shown that there are problems and challenges in the development of ML-enabled systems [2][26][33][36].”
BackgroundBabar, Z. (2024) · American Journal of Advanced Technology and Engineering Solutions · doi:10.63125/3w5cjn27 · Cited by 31
The Product Beyond the Model - An Empirical Study of Repositories of Open-Source ML Products
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00006 · Cited by 24
“Currently, academic researchers rely primarily on an outside view gathered through interviews or surveys with industry practitioners [1], [5]–[9].”
Machine Learning for Software Engineering: A Systematic MappingPreprint
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13299 · Cited by 23
Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing
Le-Cong, T., Nguyen, T.-D., Le, X.-B. D., & Murray, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3716167 · Cited by 21
Naming the Pain in machine learning-enabled systems engineering
Kalinowski, M., Méndez, D., Giray, G., Alves, A. P. S., Azevedo, K., Escovedo, T., Villamizar, H., Lopes, H., Baldassarre, M. T., Wagner, S., et al. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107866 · Cited by 20
Design of a Data Management Reference Architecture for Sustainable Agriculture
Giray, G., & Catal, C. (2021) · Sustainability · doi:10.3390/su13137309 · Cited by 17
“The performance of the deployed model should be monitored for a possible performance degradation [70,71].”
BackgroundRaffin, T., Reichenstein, T., Klier, D., Kühl, A., & Franke, J. E. (2022) · Procedia CIRP · doi:10.1016/j.procir.2022.10.063 · Cited by 18
Machine learning application development: practitioners’ insights
Rahman, M. S., Khomh, F., Hamidi, A., Cheng, J., Antoniol, G., & Washizaki, H. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09621-9 · Cited by 21
“ML engineering introduces a paradigm shift compared to conventional software engineering [8] and so the requirements engineering [7].”
BackgroundVerification and validation for trustworthy scientific machine learning
Jakeman, J. D., Barba, L. A., Martins, J. R. R. A., & O’Leary-Roseberry, T. (2026) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae59ec · Cited by 22
“Software engineering is particularly challenging in SciML models because it has “. . . all of the problems of non-ML software systems plus an additional set of ML-specific issues” [108].”
A Systematic Literature Review of Novelty Detection in Data Streams: Challenges and Opportunities
Gaudreault, J.-G., & Branco, P. (2024) · ACM Computing Surveys · doi:10.1145/3657286 · Cited by 23
“Based on other surveys in related fields of machine learning [100, 152], we define our research questions and subsequent sub-questions on the four main steps of the software development life cycle applied to the development of ND algorithms, namely: (i) Background Analysis; (ii) Requirements…”
MethodGeneralizable and interpretable learning for configuration extrapolation
Ding, Y., Pervaiz, A., Carbin, M., & Hoffmann, H. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468603 · Cited by 15
“They also hope such knowledge to generalize across different hardware and workloads [47, 51].”
BackgroundMethodDuda, S., Hofmann, P., Urbach, N., Völter, F., & Zwickel, A. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00842-7 · Cited by 19
“…engineering research suggests a re-investigation, as existing research specifically addresses certain domains and states that ‘‘on this topic, many research questions remain unanswered’’ (Giray 2021, p. 28; de Souza Nascimento et al. 2020; Shimagaki et al. 2018; Kumeno 2020; Wan et al. 2020).”
BackgroundThemes of Building LLM-Based Applications for Production: A Practitioner's View
Mailach, A., Simon, S., Dorn, J., & Siegmund, N. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00011 · Cited by 16
“Wan et al. [13] found that developers often modify traditional software engineering practices due to unique challenges present by AI, including increased efforts for collecting requirements, designing AI-enabled systems, and creating a well-suited testing dataset.”
Test & Evaluation Best Practices for Machine Learning-Enabled SystemsPreprint
Chandrasekaran, J., Cody, T., McCarthy, N., Lanus, E., & Freeman, L. E. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862589 · Cited by 14
VALIDATE: A deep dive into vulnerability prediction datasets
Esposito, M., & Falessi, D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107448 · Cited by 23
A text-based recommender system for recommending relevant news articles
Walek, B., & Müller, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125816 · Cited by 14
Chen, Y., Wan, Z., Zhuang, Y., Liu, N., Lo, D., & Yang, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705303 · Cited by 16
“…on investigating software engineering practices involved in the development of DL/ML systems, including shifts in software engineering practices [5, 96], testing methodology tailed for these systems [17, 73], misuse and updates of libraries [30, 94], refactoring practice [86], collaboration in…”
Self-citationAgility in Software 2.0 – Notebook Interfaces and MLOps with Buttresses and Rebars
Borg, M. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-030-94238-0_1 · Cited by 16
Zhang, N., Zou, Y., Xia, X., Huang, Q., Lo, D., & Li, S. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3154769 · Cited by 15
“For the two survey questions that ask for the years of experience using web APIs and the number of used web APIs, we determine the ranges based on the surveys conducted in prior studies [46], [47].”
BackgroundMethodResultSelf-citationExtracting enhanced artificial intelligence model metadata from software repositories
Tsay, J., Braz, A., Hirzel, M. J., Shinnar, A., & Mummert, T. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10206-6 · Cited by 16
“Our work is motivated by the insight that these important entangled components such as datasets are often not directly observable (echoed in other papers Wan et al. 2019; Breck et al. 2019) from software repositories.”
BackgroundMethodInfluentialHidden Feedback Loops in Machine Learning Systems: A Simulation Model and Preliminary Results
Khritankov, A. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-65854-0_5 · Cited by 18
Integrated multi-view modeling for reliable machine learning-intensive software engineering
Husen, J. H., Washizaki, H., Runpakprakun, J., Yoshioka, N., Tun, H. T., Fukazawa, Y., & Takeuchi, H. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09687-z · Cited by 16
“Extended author information available on the last page of the article Developments of ML components, which are more often referred to as ML models, are highly experimental and nondeterministic (Wan et al., 2021).”
Alamin, M. A. A., & Uddin, G. (2021) · IEEE International Conference on Autonomous Systems (ICAS) · doi:10.1109/icas49788.2021.9551151 · Cited by 22
Metadata
- Authors
- Wan, Z., Xia, X., Lo, D., & Murphy, G. C.
- Affiliations
- Zhejiang University (CN)
- University of British Columbia (CA)
- Monash University (AU)
- Singapore Management University (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2020
- Pages
- 1-1
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2970433196
- Cited by
- 181
- References
- 31
- Citations by year
- 2026: 11 · 2025: 28 · 2024: 31 · 2023: 35 · 2022: 36 · 2021: 29 · 2020: 10
- FWCI
- 23.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Basic Research Program of China (2018YFB1003904)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- machine learning systems, software development practices, software engineering process, requirements engineering, software design, software testing, skill variety, problem solving, task identity, job characteristics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.