Adoption and Effects of Software Engineering Best Practices in Machine Learning
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Background. The increasing reliance on applications with machine learning (ML) components calls for mature engineering techniques that ensure these are built in a robust and future-proof manner.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To facilitate transparency and reproducibility of the developmental process, the used computational architecture, high-performance techniques, software packages, and versioning (data, model, configurations and training scripts) should be reported13,14,16,50,64–66.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [66]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“End users should be notified clearly about any changes in the AIPM26,99, and AIPM software should have the ability to roll back to previous versions, in case an update results in significant problems20,66.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [66]
In the citing paper
“This means adequate logging and traceability of predictions and decisions is required and the AIPM interface should provide sharing of performance data with end users to enable ongoing monitoring of both individual and aggregated cases, quickly highlighting any significant deviations in performance10,26,61,66.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [66]
In the citing paper
“Deliberate stress tests like load testing, penetration testing, integration testing and unit testing are important for the verification of the AIPM from a software perspective10,26,35,46,66,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [66]
In the citing paper
“Enabling the automatic roll-back for production models is also advised to address real-time operating risks (see phase 4)66.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [66]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 1×
“ter setting aside the test sets). For all experiments we used under-sampling on the training data to remove class imbalance. We also experimented with the SMOTE over-sampling algorithm for regression [14, 53], but did not observe significant increases in performance of our models. For the grid search used for hyper-pa…”
MethodCoordinating Expertise in Software Development Teams
Faraj, S., & Sproull, L. (2000) · Management Science · doi:10.1287/mnsc.46.12.1554.12072 · Cited by 1,689
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350 · cited 2×
“est practices [2, 16, 39, 40] machine learning engineering practices [13, 44, 60] software development machine learning [31] machine learning production [42, 48] machine learning production practices [1, 5, 34, 38] machine learning deployment [18] machine learning deployment practices [43] machine learning pipelines p…”
MethodThe ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310 · cited 5×
“modified practice. Nr. Title Class Type References Rank 1 Use Sanity Checks for All External Data Sources Data N [13, 42] 22 2 Check that Input Data is Complete, Balanced and Well Distributed Data N [6, 12, 38, 42, 45] 18 3 Write Reusable Scripts for Data Cleaning and Merging Data N [1, 13, 42] 5 4 Ensure Data Labelli…”
BackgroundMethodResultInfluentialTorgo, L., Ribeiro, R. P., Pfahringer, B., & Branco, P. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40669-0_33 · Cited by 270 · cited 1×
“We also experimented with the SMOTE over-sampling algorithm for regression [14, 53], but did not observe significant increases in performance of our models.”
MethodPrinciples of survey research part 2: designing a survey
Kitchenham, B. A., & Pfleeger, S. L. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/566493.566495 · Cited by 231 · cited 2×
“Several preliminary questions were designed to specifically assign participants to groups. This renders the study a concurrent control study, in which participants are not randomly assigned to groups [33]. The target audience were teams of practitioners using ML components in a project. Specific preliminary questions…”
MethodStorey, M.-A., Zagalsky, A., Filho, F. F., Singer, L., & German, D. M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584053 · Cited by 221 · cited 1×
“ks for Production Models Deployment M [20, 44] 13 25 Log Production Predictions with the Model’s Version and Input Data Deployment M [28, 38, 49] 19 26 Use A Collaborative Development Platform Team T [10, 50] 8 27 Work Against a Shared Backlog Team T [46, 51] 9 28 Communicate, Align, and Collaborate With Multidiscipli…”
Method
Research works printed but not linked: 26
- Saleema Amershi , Andrew Begel , Christian Bird , Robert DeLine , Harald Gall , Ece Kamar , Nachiappan Nagappan , Besmira Nushi , and Thomas Zimmermann . 2019. Software engineering for machine learning: A case study . In ICSE-SEIP. IEEE , 291--300. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In ICSE-SEIP. IEEE, 291--300.
- Denis Baylor , Kevin Haas , Konstantinos Katsiapis , Sammy Leong , Rose Liu , Clemens Menwald , Hui Miao , Neoklis Polyzotis , Mitchell Trott , and Martin Zinkevich . 2019 . Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19) . 51--53. Denis Baylor, Kevin Haas, Konstantinos Katsiapis, Sammy Leong, Rose Liu, Clemens Menwald, Hui Miao, Neoklis Polyzotis, Mitchell Trott, and Martin Zinkevich. 2019. Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19). 51--53.
- Google AI Blog . 2019. Fairness Indicators . https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html. [Online ; accessed 15-12- 2019 ]. Google AI Blog. 2019. Fairness Indicators. https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html. [Online; accessed 15-12-2019].
- Google AI Blog . 2019. Responsible AI practices. https://ai.google/responsibilities/responsible-ai-practices/. [Online ; accessed 15-12- 2019 ]. Google AI Blog. 2019. Responsible AI practices. https://ai.google/responsibilities/responsible-ai-practices/. [Online; accessed 15-12-2019].
- Grady Booch and Alan W Brown . 2003. Collaborative development environments. Advances in computers 59, 1 ( 2003 ), 1--27. Grady Booch and Alan W Brown. 2003. Collaborative development environments. Advances in computers 59, 1 (2003), 1--27.
- Eric Breck , Shanqing Cai , Eric Nielsen , Michael Salib , and D. Sculley . 2016. What's your ML test score? A rubric for ML production systems . In Reliable Machine Learning in the Wild - NeurIPS Workshop. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2016. What's your ML test score? A rubric for ML production systems. In Reliable Machine Learning in the Wild - NeurIPS Workshop.
- Marcus Ciolkowski , Oliver Laitenberger , Sira Vegas , and Stefan Biffl . 2003. Practical experiences in the design and conduct of surveys in empirical software engineering . In Empirical methods and studies in software engineering . Springer , 104--128. Marcus Ciolkowski, Oliver Laitenberger, Sira Vegas, and Stefan Biffl. 2003. Practical experiences in the design and conduct of surveys in empirical software engineering. In Empirical methods and studies in software engineering. Springer, 104--128.
- Cloudfactory. 2019. The ultimate guide to data labeling for ML. https://www.cloudfactory.com/data-labeling-guide. [Online ; accessed 15-12- 2019 ]. Cloudfactory. 2019. The ultimate guide to data labeling for ML. https://www.cloudfactory.com/data-labeling-guide. [Online; accessed 15-12-2019].
- Elizamary de Souza Nascimento , Iftekhar Ahmed , Edson Oliveira , Márcio Piedade Palheta , Igor Steinmacher, and Tayana Conte. 2019 . Understanding Development Process of Machine Learning Systems: Challenges and Solutions. In ESEM. IEEE , 1--6. Elizamary de Souza Nascimento, Iftekhar Ahmed, Edson Oliveira, Márcio Piedade Palheta, Igor Steinmacher, and Tayana Conte. 2019. Understanding Development Process of Machine Learning Systems: Challenges and Solutions. In ESEM. IEEE, 1--6.
- Ted Dunning and Ellen Friedman . 2019. Machine Learning Logistics. https://mapr.com/ebook/machine-learning-logistics/. [Online ; accessed 15-12- 2019 ]. Ted Dunning and Ellen Friedman. 2019. Machine Learning Logistics. https://mapr.com/ebook/machine-learning-logistics/. [Online; accessed 15-12-2019].
- Matthias Feurer , Aaron Klein , Katharina Eggensperger , Jost Springenberg , Manuel Blum , and Frank Hutter . 2015. Efficient and Robust Automated Machine Learning . In NeurIPS 2015 . 2962--2970. Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. In NeurIPS 2015. 2962--2970.
- Alexandre Fréchette , Lars Kotthoff , Tomasz P. Michalak , Talal Rahwan , Holger H. Hoos , and Kevin Leyton-Brown . 2016 . Using the Shapley Value to Analyze Algorithm Portfolios . In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16) . AAAI Press, 3397--3403. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12495 Alexandre Fréchette, Lars Kotthoff, Tomasz P. Michalak, Talal Rahwan, Holger H. Hoos, and Kevin Leyton-Brown. 2016. Using the Shapley Value to Analyze Algorithm Portfolios. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). AAAI Press, 3397--3403. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12495
- Vahid Garousi , Michael Felderer , Mika V Mäntylä , and Austen Rainer . 2019. Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038 ( 2019 ). Vahid Garousi, Michael Felderer, Mika V Mäntylä, and Austen Rainer. 2019. Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038 (2019). · arXiv
- Scott M Lundberg , Gabriel Erion , Hugh Chen , Alex DeGrave , Jordan M Prutkin , Bala Nair , Ronit Katz , Jonathan Himmelfarb , Nisha Bansal , and Su-In Lee . 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 ( 2020 ), 2522--5839. Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 (2020), 2522--5839.
- V.M. Megler . 2019. Managing Machine Learning Projects. https://d1.awsstatic.com/whitepapers/aws-managing-ml-projects.pdf. [Online ; accessed 15-12- 2019 ]. V.M. Megler. 2019. Managing Machine Learning Projects. https://d1.awsstatic.com/whitepapers/aws-managing-ml-projects.pdf. [Online; accessed 15-12-2019].
- Yuji Roh , Geon Heo , and Steven Euijong Whang . 2019. A survey on data collection for machine learning: a big data-ai integration perspective . IEEE Transactions on Knowledge and Data Engineering ( 2019 ). Yuji Roh, Geon Heo, and Steven Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering (2019).
- Bernd Rohrmann . 2007. Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data . University of Melbourne , Australia ( 2007 ). Bernd Rohrmann. 2007. Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data. University of Melbourne, Australia (2007).
- Alkis Polyzotis Martin A. Zinkevich Steven Whang Sudip Roy . 2017 . Data management challenges in production machine learning. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45a9dcf23dbdfa24dbced358f825636c58518afa.pdf. [Online; accessed 15-12-2019]. Alkis Polyzotis Martin A. Zinkevich Steven Whang Sudip Roy. 2017. Data management challenges in production machine learning. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45a9dcf23dbdfa24dbced358f825636c58518afa.pdf. [Online; accessed 15-12-2019].
- Carlton Sapp . 2017. Preparing and Architecting for Machine Learning. https://www.gartner.com. [Online ; accessed 15-12- 2019 ]. Carlton Sapp. 2017. Preparing and Architecting for Machine Learning. https://www.gartner.com. [Online; accessed 15-12-2019].
- Danilo Sato , Arif Wider , and Christoph Windheuser . 2019. Continuous Delivery for Machine Learning. https://martinfowler.com/articles/cd4ml.html. [Online ; accessed 15-12- 2019 ]. Danilo Sato, Arif Wider, and Christoph Windheuser. 2019. Continuous Delivery for Machine Learning. https://martinfowler.com/articles/cd4ml.html. [Online; accessed 15-12-2019].
- David Sculley , Gary Holt , Daniel Golovin , Eugene Davydov , Todd Phillips , Dietmar Ebner , Vinay Chaudhary , Michael Young , Jean-Francois Crespo , and Dan Dennison . 2015. Hidden technical debt in machine learning systems . In NeurIPS 2015 . 2503--2511. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In NeurIPS 2015. 2503--2511.
- Todd Sedano , Paul Ralph , and Cécile Péraire . 2019. The product backlog . In ICSE. IEEE , 200--211. Todd Sedano, Paul Ralph, and Cécile Péraire. 2019. The product backlog. In ICSE. IEEE, 200--211.
- Vinay Sridhar , Sriram Subramanian , Dulcardo Arteaga , Swaminathan Sundararaman , Drew Roselli , and Nisha Talagala . 2018 . Model governance: Reducing the anarchy of production ML . In 2018 USENIX Annual Technical Conference (USENIX ATC 18) . 351--358. Vinay Sridhar, Sriram Subramanian, Dulcardo Arteaga, Swaminathan Sundararaman, Drew Roselli, and Nisha Talagala. 2018. Model governance: Reducing the anarchy of production ML. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 351--358.
- Jeff Sutherland and Ken Schwaber . 2013. The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268 ( 2013 ). Jeff Sutherland and Ken Schwaber. 2013. The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268 (2013).
- Joost Visser , Sylvan Rigal , Gijs Wijnholds , Pascal van Eck , and Rob van der Leek . 2016. Building Maintainable Software , C# Edition: Ten Guidelines for Future-Proof Code . O'Reilly Media, Inc. Joost Visser, Sylvan Rigal, Gijs Wijnholds, Pascal van Eck, and Rob van der Leek. 2016. Building Maintainable Software, C# Edition: Ten Guidelines for Future-Proof Code. O'Reilly Media, Inc.
- Rüdiger Wirth and Jochen Hipp . 2000 . CRISP-DM: Towards a standard process model for data mining . In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39 . Rüdiger Wirth and Jochen Hipp. 2000. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39.
Cited by
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
“…logging and traceability of predictions and decisions is required and the AIPM interface should provide sharing of performance data with end users to enable ongoing monitoring of both individual and aggregated cases, quickly highlighting any signi fi cant deviations in performance 10,26,61,66 .”
BackgroundMethodInfluentialSoftware Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
“[222] 138 studies (2007-2019) correctness, model relevance, robustness, security, data privacy, efficiency, fairness, interpretability testing workflow, testing components, testing properties, application scenarios 4 testing challenges categories: test input generation, test assessment criteria, oracle problem, testin…”
BackgroundArtificial Intelligence Governance For Businesses
Schneider, J., Abraham, R., Meske, C., & Vom Brocke, J. (2023) · Information Systems Management · doi:10.1080/10580530.2022.2085825 · Cited by 164
“Both should be shareable and employable in various areas of a business (Serban et al., 2020).”
BackgroundMethodInfluentialNahar, N., Zhou, S., Lewis, G., & Kästner, C. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510209 · Cited by 131
“Much attention in practice recently focuses on building robust ML pipelines for training and deploying models in a scalable fashion, often under names such as “AI engineering,” “SysML,” and “MLOps” [51, 59, 66, 89].”
BackgroundMethodInfluentialGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metadata
- Authors
- Serban, A., van der Blom, K., Hoos, H., & Visser, J.
- Affiliations
- ICIS, Radboud University, Software Improvement Group, Amsterdam, Netherlands
- LIACS, Leiden University, Leiden, Netherlands
- Leiden University (NL)
- Venue
- Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)
- Publisher
- ACM
- Published
- 2020-10-05
- Pages
- 1-12
- ISSN
- 1938-6451 (ISSN-L)
- OpenAlex
- W3045719304
- Cited by
- 127
- References
- 17
- Citations by year
- 2026: 17 · 2025: 18 · 2024: 19 · 2023: 22 · 2022: 32 · 2021: 16 · 2020: 3
- FWCI
- 13.84
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- software engineering best practices, machine learning engineering, software maturity, robustness, machine learning applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Adoption and Effects of Software Engineering Best Practices in Machine Learning. https://app.cochat.ai/papers/10.1145/3382494.3410681
- The paper
- Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020). Adoption and Effects of Software Engineering Best Practices in Machine Learning. Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). https://doi.org/10.1145/3382494.3410681
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.