How does Machine Learning Change Software Development Practices?
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Adding an ability for a system to learn inherently adds uncertainty into the system. Given the rising popularity of incorporating machine learning into systems, we wondered how the addition alters software development practices. We performed a mixture of qualitative and quantitative studies with 14 interviewees and 342 survey respondents from 26 countries across four continents to elicit significant differences between the development of machine learning systems and the development of non-machine-learning systems. Our study uncovers significant differences in various aspects of software engineering (e.g., requirements, design, testing, and process) and work characteristics (e.g., skill variety, problem solving and task identity). Based on our findings, we highlight future research directions and provide recommendations for practitioners.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is complex for practitioners to comprehend the scopes and impacts of architectural changes [95, 103].”
From Software Architecture in Practice: Challenges and Opportunities (2023) · cites this paper as [103]
In the citing paper
“Moreover, software engineering has had decades of development in data management, which could be readily adapted to AI practices in many cases [190].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [190]
In the citing paper
“Exploratory processes in ML share many similarities to scientific programming, and would benefit from lessons learned in embracing uncertainty [190].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [190]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · cited 1×
“We conducted a thematic analysis [8] to process the recorded interviews by following the steps below: Transcribing and Coding.”
MethodA Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“We then use the Cohen’s Kappa measure [10] to examine the agreement between the two labelers.”
MethodHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808 · cited 1×
“[15], conducting 12 to 15 interviews of a homogeneous group is adequate to reach saturation.”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 2×
“Machine learning (ML) has progressed dramatically over the past three decades, from a laboratory curiosity to a practical technology in widespread commercial use [19].”
BackgroundHumphrey, S. E., Nahrgang, J. D., & Morgeson, F. P. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.5.1332 · Cited by 2,303 · cited 1×
“[18] in applied psychology, which consists of 21 work characteristics, e.”
BackgroundProbabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995 · cited 1×
“Uncertainty plays a fundamental role in probabilistic modeling [14]: Observed data can be consistent with various models, and thus which model is appropriate given the data is uncertain.”
BackgroundDeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
MethodSoftware Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 3×
“Most similar to our study is an empirical investigation of integrating AI capabilities into software and services and best practices from Microsoft teams [4].”
BackgroundMethodInfluential
Research works printed but not linked: 3
- romano, 2006, Appropriate statistics for ordinal level data: Should we really be using t-test and cohen'sd for evaluating group differences on the nsse and other surveys, Proc Annu Meet Florida Assoc Institutional Res, 1
- 10.1007/bf02729815 · doi:10.1007/bf02729815
- ng, 2017, Machine Learning Yearning
Cited by
Software Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
“com [21], AI model criteria relevant for end users [56], automatic version control in notebooks [98], or practices collected via practitioner surveys and/or interviews [203, 224].”
BackgroundThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Lo, S. K., Lu, Q., Wang, C., Paik, H.-Y., & Zhu, L. (2022) · ACM Computing Surveys · doi:10.1145/3450288 · Cited by 202
“olves a high number of participating clients’ collaboration which forms a large-scale distributed system. This calls for the software engineering thinking apart from the core machine learning concept [164]. Hence, the second motivation is to explore how to develop federated learning systems by examining existing studi…”
MethodNahar, N., Zhou, S., Lewis, G., & Kästner, C. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510209 · Cited by 131
“Reported problems relate to poor code quality (P1b, P2a, P3b, P5a, P6a-b, P10a, P11a, P14a, P15b-c, P17a, P18a, P19a, P20a-b, P26a) [9, 27, 34, 37, 74, 86, 105], insufficient documentation (P5a-b, P6a-b, P10a, P15c, P26a) [8, 46, 64, 113], and not extending version control to data and models (P3c, P7a, P10a, P14a, P20…”
BackgroundResultInfluential
Metadata
- Authors
- Wan, Z., Xia, X., Lo, D., & Murphy, G. C.
- Affiliations
- Zhejiang University (CN)
- University of British Columbia (CA)
- Monash University (AU)
- Singapore Management University (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2020
- Pages
- 1-1
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2970433196
- Cited by
- 181
- References
- 31
- Citations by year
- 2026: 11 · 2025: 28 · 2024: 31 · 2023: 35 · 2022: 36 · 2021: 29 · 2020: 10
- FWCI
- 23.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Basic Research Program of China (2018YFB1003904)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- machine learning systems, software development practices, software engineering process, requirements engineering, software design, software testing, skill variety, problem solving, task identity, job characteristics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How does Machine Learning Change Software Development Practices? https://app.cochat.ai/papers/10.1109/tse.2019.2937083
- The paper
- Wan, Z., Xia, X., Lo, D., & Murphy, G. C. (2020). How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2019.2937083
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.