CoChat

Adoption and Effects of Software Engineering Best Practices in Machine Learning

Open access
Serban, Alex, van der Blom, Koen, Hoos, Holger, Visser, Joost · 2020 · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.

Abstract

Background. The increasing reliance on applications with machine learning (ML) components calls for mature engineering techniques that ensure these are built in a robust and future-proof manner.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3382494.3410681

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 19 references

Research works printed but not linked: 26

  1. Saleema Amershi , Andrew Begel , Christian Bird , Robert DeLine , Harald Gall , Ece Kamar , Nachiappan Nagappan , Besmira Nushi , and Thomas Zimmermann . 2019. Software engineering for machine learning: A case study . In ICSE-SEIP. IEEE , 291--300. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In ICSE-SEIP. IEEE, 291--300.
  2. Denis Baylor , Kevin Haas , Konstantinos Katsiapis , Sammy Leong , Rose Liu , Clemens Menwald , Hui Miao , Neoklis Polyzotis , Mitchell Trott , and Martin Zinkevich . 2019 . Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19) . 51--53. Denis Baylor, Kevin Haas, Konstantinos Katsiapis, Sammy Leong, Rose Liu, Clemens Menwald, Hui Miao, Neoklis Polyzotis, Mitchell Trott, and Martin Zinkevich. 2019. Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19). 51--53.
  3. Google AI Blog . 2019. Fairness Indicators . https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html. [Online ; accessed 15-12- 2019 ]. Google AI Blog. 2019. Fairness Indicators. https://ai.googleblog.com/2019/12/fairness-indicators-scalable.html. [Online; accessed 15-12-2019].
  4. Google AI Blog . 2019. Responsible AI practices. https://ai.google/responsibilities/responsible-ai-practices/. [Online ; accessed 15-12- 2019 ]. Google AI Blog. 2019. Responsible AI practices. https://ai.google/responsibilities/responsible-ai-practices/. [Online; accessed 15-12-2019].
  5. Grady Booch and Alan W Brown . 2003. Collaborative development environments. Advances in computers 59, 1 ( 2003 ), 1--27. Grady Booch and Alan W Brown. 2003. Collaborative development environments. Advances in computers 59, 1 (2003), 1--27.
  6. Eric Breck , Shanqing Cai , Eric Nielsen , Michael Salib , and D. Sculley . 2016. What's your ML test score? A rubric for ML production systems . In Reliable Machine Learning in the Wild - NeurIPS Workshop. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2016. What's your ML test score? A rubric for ML production systems. In Reliable Machine Learning in the Wild - NeurIPS Workshop.
  7. Marcus Ciolkowski , Oliver Laitenberger , Sira Vegas , and Stefan Biffl . 2003. Practical experiences in the design and conduct of surveys in empirical software engineering . In Empirical methods and studies in software engineering . Springer , 104--128. Marcus Ciolkowski, Oliver Laitenberger, Sira Vegas, and Stefan Biffl. 2003. Practical experiences in the design and conduct of surveys in empirical software engineering. In Empirical methods and studies in software engineering. Springer, 104--128.
  8. Cloudfactory. 2019. The ultimate guide to data labeling for ML. https://www.cloudfactory.com/data-labeling-guide. [Online ; accessed 15-12- 2019 ]. Cloudfactory. 2019. The ultimate guide to data labeling for ML. https://www.cloudfactory.com/data-labeling-guide. [Online; accessed 15-12-2019].
  9. Elizamary de Souza Nascimento , Iftekhar Ahmed , Edson Oliveira , Márcio Piedade Palheta , Igor Steinmacher, and Tayana Conte. 2019 . Understanding Development Process of Machine Learning Systems: Challenges and Solutions. In ESEM. IEEE , 1--6. Elizamary de Souza Nascimento, Iftekhar Ahmed, Edson Oliveira, Márcio Piedade Palheta, Igor Steinmacher, and Tayana Conte. 2019. Understanding Development Process of Machine Learning Systems: Challenges and Solutions. In ESEM. IEEE, 1--6.
  10. Ted Dunning and Ellen Friedman . 2019. Machine Learning Logistics. https://mapr.com/ebook/machine-learning-logistics/. [Online ; accessed 15-12- 2019 ]. Ted Dunning and Ellen Friedman. 2019. Machine Learning Logistics. https://mapr.com/ebook/machine-learning-logistics/. [Online; accessed 15-12-2019].
  11. Matthias Feurer , Aaron Klein , Katharina Eggensperger , Jost Springenberg , Manuel Blum , and Frank Hutter . 2015. Efficient and Robust Automated Machine Learning . In NeurIPS 2015 . 2962--2970. Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. In NeurIPS 2015. 2962--2970.
  12. Alexandre Fréchette , Lars Kotthoff , Tomasz P. Michalak , Talal Rahwan , Holger H. Hoos , and Kevin Leyton-Brown . 2016 . Using the Shapley Value to Analyze Algorithm Portfolios . In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16) . AAAI Press, 3397--3403. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12495 Alexandre Fréchette, Lars Kotthoff, Tomasz P. Michalak, Talal Rahwan, Holger H. Hoos, and Kevin Leyton-Brown. 2016. Using the Shapley Value to Analyze Algorithm Portfolios. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). AAAI Press, 3397--3403. http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12495
  13. Vahid Garousi , Michael Felderer , Mika V Mäntylä , and Austen Rainer . 2019. Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038 ( 2019 ). Vahid Garousi, Michael Felderer, Mika V Mäntylä, and Austen Rainer. 2019. Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038 (2019). · arXiv
  14. Scott M Lundberg , Gabriel Erion , Hugh Chen , Alex DeGrave , Jordan M Prutkin , Bala Nair , Ronit Katz , Jonathan Himmelfarb , Nisha Bansal , and Su-In Lee . 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 ( 2020 ), 2522--5839. Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 (2020), 2522--5839.
  15. V.M. Megler . 2019. Managing Machine Learning Projects. https://d1.awsstatic.com/whitepapers/aws-managing-ml-projects.pdf. [Online ; accessed 15-12- 2019 ]. V.M. Megler. 2019. Managing Machine Learning Projects. https://d1.awsstatic.com/whitepapers/aws-managing-ml-projects.pdf. [Online; accessed 15-12-2019].
  16. Yuji Roh , Geon Heo , and Steven Euijong Whang . 2019. A survey on data collection for machine learning: a big data-ai integration perspective . IEEE Transactions on Knowledge and Data Engineering ( 2019 ). Yuji Roh, Geon Heo, and Steven Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering (2019).
  17. Bernd Rohrmann . 2007. Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data . University of Melbourne , Australia ( 2007 ). Bernd Rohrmann. 2007. Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data. University of Melbourne, Australia (2007).
  18. Alkis Polyzotis Martin A. Zinkevich Steven Whang Sudip Roy . 2017 . Data management challenges in production machine learning. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45a9dcf23dbdfa24dbced358f825636c58518afa.pdf. [Online; accessed 15-12-2019]. Alkis Polyzotis Martin A. Zinkevich Steven Whang Sudip Roy. 2017. Data management challenges in production machine learning. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45a9dcf23dbdfa24dbced358f825636c58518afa.pdf. [Online; accessed 15-12-2019].
  19. Carlton Sapp . 2017. Preparing and Architecting for Machine Learning. https://www.gartner.com. [Online ; accessed 15-12- 2019 ]. Carlton Sapp. 2017. Preparing and Architecting for Machine Learning. https://www.gartner.com. [Online; accessed 15-12-2019].
  20. Danilo Sato , Arif Wider , and Christoph Windheuser . 2019. Continuous Delivery for Machine Learning. https://martinfowler.com/articles/cd4ml.html. [Online ; accessed 15-12- 2019 ]. Danilo Sato, Arif Wider, and Christoph Windheuser. 2019. Continuous Delivery for Machine Learning. https://martinfowler.com/articles/cd4ml.html. [Online; accessed 15-12-2019].
  21. David Sculley , Gary Holt , Daniel Golovin , Eugene Davydov , Todd Phillips , Dietmar Ebner , Vinay Chaudhary , Michael Young , Jean-Francois Crespo , and Dan Dennison . 2015. Hidden technical debt in machine learning systems . In NeurIPS 2015 . 2503--2511. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In NeurIPS 2015. 2503--2511.
  22. Todd Sedano , Paul Ralph , and Cécile Péraire . 2019. The product backlog . In ICSE. IEEE , 200--211. Todd Sedano, Paul Ralph, and Cécile Péraire. 2019. The product backlog. In ICSE. IEEE, 200--211.
  23. Vinay Sridhar , Sriram Subramanian , Dulcardo Arteaga , Swaminathan Sundararaman , Drew Roselli , and Nisha Talagala . 2018 . Model governance: Reducing the anarchy of production ML . In 2018 USENIX Annual Technical Conference (USENIX ATC 18) . 351--358. Vinay Sridhar, Sriram Subramanian, Dulcardo Arteaga, Swaminathan Sundararaman, Drew Roselli, and Nisha Talagala. 2018. Model governance: Reducing the anarchy of production ML. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 351--358.
  24. Jeff Sutherland and Ken Schwaber . 2013. The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268 ( 2013 ). Jeff Sutherland and Ken Schwaber. 2013. The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268 (2013).
  25. Joost Visser , Sylvan Rigal , Gijs Wijnholds , Pascal van Eck , and Rob van der Leek . 2016. Building Maintainable Software , C# Edition: Ten Guidelines for Future-Proof Code . O'Reilly Media, Inc. Joost Visser, Sylvan Rigal, Gijs Wijnholds, Pascal van Eck, and Rob van der Leek. 2016. Building Maintainable Software, C# Edition: Ten Guidelines for Future-Proof Code. O'Reilly Media, Inc.
  26. Rüdiger Wirth and Jochen Hipp . 2000 . CRISP-DM: Towards a standard process model for data mining . In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39 . Rüdiger Wirth and Jochen Hipp. 2000. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39.

Cited by

See all 155

Metadata

Authors
Serban, A., van der Blom, K., Hoos, H., & Visser, J.
Affiliations
  • ICIS, Radboud University, Software Improvement Group, Amsterdam, Netherlands
  • LIACS, Leiden University, Leiden, Netherlands
  • Leiden University (NL)
Venue
Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)
Publisher
ACM
Published
2020-10-05
Pages
1-12
ISSN
1938-6451 (ISSN-L)
OpenAlex
W3045719304
Cited by
127
References
17
Citations by year
2026: 17 · 2025: 18 · 2024: 19 · 2023: 22 · 2022: 32 · 2021: 16 · 2020: 3
FWCI
13.84
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Leiden Repository (Leiden University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
software engineering best practices, machine learning engineering, software maturity, robustness, machine learning applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Adoption and Effects of Software Engineering Best Practices in Machine Learning. https://app.cochat.ai/papers/10.1145/3382494.3410681
The paper
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020). Adoption and Effects of Software Engineering Best Practices in Machine Learning. Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). https://doi.org/10.1145/3382494.3410681

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.