How does Machine Learning Change Software Development Practices?
The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Adding an ability for a system to learn inherently adds uncertainty into the system. Given the rising popularity of incorporating machine learning into systems, we wondered how the addition alters software development practices. We performed a mixture of qualitative and quantitative studies with 14 interviewees and 342 survey respondents from 26 countries across four continents to elicit significant differences between the development of machine learning systems and the development of non-machine-learning systems. Our study uncovers significant differences in various aspects of software engineering (e.g., requirements, design, testing, and process) and work characteristics (e.g., skill variety, problem solving and task identity). Based on our findings, we highlight future research directions and provide recommendations for practitioners.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is complex for practitioners to comprehend the scopes and impacts of architectural changes [95, 103].”
From Software Architecture in Practice: Challenges and Opportunities (2023) · cites this paper as [103]
In the citing paper
“Moreover, software engineering has had decades of development in data management, which could be readily adapted to AI practices in many cases [190].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [190]
In the citing paper
“Exploratory processes in ML share many similarities to scientific programming, and would benefit from lessons learned in embracing uncertainty [190].”
From Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers (2023) · cites this paper as [190]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · cited 1×
“We conducted a thematic analysis [8] to process the recorded interviews by following the steps below: Transcribing and Coding.”
MethodA Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“We then use the Cohen’s Kappa measure [10] to examine the agreement between the two labelers.”
MethodHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808 · cited 1×
“[15], conducting 12 to 15 interviews of a homogeneous group is adequate to reach saturation.”
BackgroundTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 2×
“Machine learning (ML) has progressed dramatically over the past three decades, from a laboratory curiosity to a practical technology in widespread commercial use [19].”
BackgroundHumphrey, S. E., Nahrgang, J. D., & Morgeson, F. P. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.5.1332 · Cited by 2,303 · cited 1×
“[18] in applied psychology, which consists of 21 work characteristics, e.”
BackgroundProbabilistic machine learning and artificial intelligence
Ghahramani, Z. (2015) · Nature · doi:10.1038/nature14541 · Cited by 1,995 · cited 1×
“Uncertainty plays a fundamental role in probabilistic modeling [14]: Observed data can be consistent with various models, and thus which model is appropriate given the data is uncertain.”
BackgroundDeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
MethodSoftware Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 3×
“Most similar to our study is an empirical investigation of integrating AI capabilities into software and services and best practices from Microsoft teams [4].”
BackgroundMethodInfluentialA Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968 · cited 1×
“Data collection literature focuses on three lines of research [31]: data acquisition techniques to discover,”
BackgroundMaintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
DeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
Kitchenham, B. A., & Pfleeger, S. L. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_3 · Cited by 484 · cited 1×
“We followed Kitchenham and Pfleeger’s guidelines for personal opinion surveys [23] and used an anonymous survey to increase response rates [37].”
MethodGuide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0
Bourque, P., & Fairley, R. E. (2014) · IEEE Computer Society Press eBooks · Cited by 456
Enterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 439 · cited 1×
“company via interviews to identify pain points from a general tooling perspective [13], and explore challenges and barriers for adopting visual analytic tools [20].”
BackgroundTesting and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 389 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
MethodInteractions with big data analytics
Fisher, D., DeLine, R., Czerwinski, M., & Drucker, S. (2012) · Interactions · doi:10.1145/2168931.2168943 · Cited by 361 · cited 1×
“company via interviews to identify pain points from a general tooling perspective [13], and explore challenges and barriers for adopting visual analytic tools [20].”
BackgroundTFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
MethodThe emerging role of data scientists on software development teams
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884783 · Cited by 231 · cited 1×
“interviewed sixteen data scientists at Microsoft to identify five working styles [21], and supplement Harris et al.”
BackgroundMODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
Data scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
Estimating conversion rate in display advertising from past erformance data
Lee, K.-C., Orten, B., Dasdan, A., & Li, W. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339651 · Cited by 197
Software Development Environments for Scientific and Engineering Software: A Series of Case Studies
Carver, J. C., Kendall, R. P., Squires, S. E., & Post, D. E. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.77 · Cited by 166 · cited 1×
“The exploratory process of preliminary experiments in ML development is similar to scientific programming [9] and may benefit from the lessons learned from scientific programming.”
BackgroundOn Challenges in Machine Learning Model Management
Schelter, S., Bießmann, F., Januschowski, T., Salinas, D., Seufert, S., & Szarvas, G. (2015) · IEEE Data(base) Engineering Bulletin · Cited by 139
Continuously Reproducing Toolchains in Pattern Recognition and Machine Learning Experiments
Anjos, A., Günther, M., Pereira, T. D. F., Korshunov, P., Mohammadi, A., & Marcel, S. (2017) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 76
BigDebug: debugging primitives for interactive big data processing in spark
Gulzar, M. A., Interlandi, M., Yoo, S., Tetali, S. D., Condie, T., Millstein, T. D., & Kim, M. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884813 · Cited by 69
Analyzing the Analyzers: An Introspective Survey of Data Scientists and Their Work
Harris, H. A. H., Murphy, S. P., & Vaisman, M. (2013) · CERN Document Server (European Organization for Nuclear Research) · Cited by 61
The Effects of Appeals, Anonymity, and Feedback on Mail Survey Response Patterns from Salespeople
Tyagi, P. K. (1989) · Journal of the Academy of Marketing Science · doi:10.1177/009207038901700305 · Cited by 60 · cited 1×
“We followed Kitchenham and Pfleeger’s guidelines for personal opinion surveys [23] and used an anonymous survey to increase response rates [37].”
MethodLAMP: data provenance for graph based machine learning algorithms through derivative computation
Ma, S., Aafer, Y., Xu, Z., Lee, W., Zhai, J., Liu, Y., & Zhang, X. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106291 · Cited by 20 · cited 1×
“To tackle the ML specific issues, recent studies have put effort into building tools for testing [26], [30], [36], [39] and debugging [16], [27], [28] of machine learning code, and creating frameworks and environments to support development of ML systems [3], [6].”
Large-scale empirical study on machine learning related questions on Stack Overflow
Wan, Z., Tao, J.-H., Liang, J.-K., Cai, Z.-G., Chang, C., Qiao, L., & Zhou, Q.-N. (2019) · Journal of Zhejiang University(Engineering Science) · doi:10.3785/j.issn.1008-973x.2019.05.001 · Cited by 3 · cited 1×
“Some investigated machine learning related questions on Stack Overflow [38].”
BackgroundSelf-citation
Research works printed but not linked: 3
- romano, 2006, Appropriate statistics for ordinal level data: Should we really be using t-test and cohen'sd for evaluating group differences on the nsse and other surveys, Proc Annu Meet Florida Assoc Institutional Res, 1
- 10.1007/bf02729815 · doi:10.1007/bf02729815
- ng, 2017, Machine Learning Yearning
Cited by
Software Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
“com [21], AI model criteria relevant for end users [56], automatic version control in notebooks [98], or practices collected via practitioner surveys and/or interviews [203, 224].”
BackgroundThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Lo, S. K., Lu, Q., Wang, C., Paik, H.-Y., & Zhu, L. (2022) · ACM Computing Surveys · doi:10.1145/3450288 · Cited by 202
“olves a high number of participating clients’ collaboration which forms a large-scale distributed system. This calls for the software engineering thinking apart from the core machine learning concept [164]. Hence, the second motivation is to explore how to develop federated learning systems by examining existing studi…”
MethodNahar, N., Zhou, S., Lewis, G., & Kästner, C. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510209 · Cited by 131
“Reported problems relate to poor code quality (P1b, P2a, P3b, P5a, P6a-b, P10a, P11a, P14a, P15b-c, P17a, P18a, P19a, P20a-b, P26a) [9, 27, 34, 37, 74, 86, 105], insufficient documentation (P5a-b, P6a-b, P10a, P15c, P26a) [8, 46, 64, 113], and not extending version control to data and models (P3c, P7a, P10a, P14a, P20…”
BackgroundResultInfluential
Metadata
- Authors
- Wan, Z., Xia, X., Lo, D., & Murphy, G. C.
- Affiliations
- Zhejiang University (CN)
- University of British Columbia (CA)
- Monash University (AU)
- Singapore Management University (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2020
- Pages
- 1-1
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2970433196
- Cited by
- 181
- References
- 31
- Citations by year
- 2026: 11 · 2025: 28 · 2024: 31 · 2023: 35 · 2022: 36 · 2021: 29 · 2020: 10
- FWCI
- 23.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Basic Research Program of China (2018YFB1003904)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- machine learning systems, software development practices, software engineering process, requirements engineering, software design, software testing, skill variety, problem solving, task identity, job characteristics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.