CoChat
Sign Up
Cited by 256Explainable Artificial Intelligence (XAI)Safety Systems Engineering in AutonomySoftware Testing and Debugging Techniques

Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges

Open access
Ashmore, Rob, Calinescu, Radu, Paterson, Colin·2022·ACM Computing Surveys
DOI 10.1145/3453444
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning has evolved into an enabling technology for a wide range of highly successful applications. The potential for this success to continue and accelerate has placed machine learning (ML) at the top of research, economic, and political agendas. Such unprecedented interest is fuelled by a vision of ML applicability extending to healthcare, transportation, defence, and other domains of great societal importance. Achieving this vision requires the use of ML in safety-critical applications that demand levels of assurance beyond those needed for current ML applications. Our article provides a comprehensive survey of the state of the art in the assurance of ML , i.e., in the generation of evidence that ML is sufficiently safe for its intended use. The survey covers the methods capable of providing such evidence at different stages of the machine learning lifecycle , i.e., of the complex, iterative process that starts with the collection of the data used to train an ML component for a system, and ends with the deployment of that component within the system. The article begins with a systematic presentation of the ML lifecycle and its stages. We then define assurance desiderata for each stage, review existing methods that contribute to achieving these desiderata, and identify open challenges that require further research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3453444

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multi-dataset-multi-model pipelines, where existing ML models are used for pre-processing data or training new ML models, are becoming increasingly common.

    “Other scenarios that are becoming increasingly common are multi-dataset-multi-model pipelines, where existing ML models are used for pre-processing data or training new ML models [5].”

    From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [5]

    Section 5.2.4 says an existing model can provide a starting point for training another, and Section 2 discusses preprocessing data collected from multiple sources. Neither passage says that existing ML models perform the preprocessing or that multi-dataset, multi-model pipelines are becoming increasingly common. Page numbers were not provided in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The steps of model building and testing are frequently followed by the need to collect new data, or enriching the existing dataset.

    “The first of these relates to data iteration, where the steps of model building and testing are frequently followed by the need to collect new data, or enriching the existing dataset [5,11,31,34].”

    From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [5]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 122 · Cited by 256

Research works printed but not linked: 46

  1. [8]8. D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz. 2012. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In International Workshop on Ambient Assisted Living. 216ś223.
  2. [13]13. Rob Ashmore and Elizabeth Lennon. 2017. Progress Towards the Assurance of Non-Traditional Software. In Develop- ments in System Safety Engineering, 25th Safety-Critical Systems Symposium. 33ś48.
  3. [14]14. Rob Ashmore and Bhopinder Madahar. 2019. Rethinking Diversity in the Context of Autonomous Systems. In Engineering Safe Autonomy, 27th Safety-Critical Systems Symposium. 175ś192.
  4. [21]21. Robin Bloomfield and Peter Bishop. 2010. Safety and assurance cases: Past, present and possible futureśan Adelard perspective. In Making Systems Safer. Springer, 51ś67.
  5. [26]26. Carla E Brodley and Mark A Friedl. 1999. Identifying mislabeled training data. Journal of Artificial Intelligence Research 11 (1999), 131ś167.
  6. [33]33. Nitesh V Chawla, Aleksandar Lazarevic, Lawrence O Hall, and Kevin W Bowyer. 2003. SMOTEBoost: Improving prediction of the minority class in boosting. In European Conference on Principles of Data Mining and Knowledge Discovery. 107ś119.
  7. [34]34. Liming Chen and Algirdas Avizienis. 1978. N-version programming: A fault-tolerance approach to reliability of software operation. In 8th IEEE International Symposium on Fault-Tolerant Computing, Vol. 1. 3ś9.
  8. [38]38. David A Cieslak and Nitesh V Chawla. 2009. A framework for monitoring classifiers performance: when and why failure occurs? Knowledge and Information Systems 18, 1 (2009), 83ś108.
  9. [45]45. Tommaso Dreossi, Somesh Jha, and Sanjit A Seshia. 2018. Semantic adversarial deep learning. arXiv:1804.07045 · arXiv
  10. [50]50. Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2015. Fundamental limits on adversarial robustness. In ICML Workshop on Deep Learning.
  11. [51]51. Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2016. Robustness of Classifiers: From Adversarial to Random Noise (NIPS'16).
  12. [60]60. Aurélien Géron. 2017. Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems. " O'Reilly Media, Inc. ".
  13. [61]61. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT Press.
  14. [75]75. Bandar Seri Iskandar. 2017. Terrorism detection based on sentiment analysis using machine learning. Journal of Engineering and Applied Sciences 12, 3 (2017), 691ś698.
  15. [82]82. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification. Springer, 97ś117.
  16. [87]87. Udayan Khurana, Horst Samulowitz, and Deepak Turaga. 2018. Feature engineering for predictive modeling using reinforcement learning. In 32nd AAAI Conference on Artificial Intelligence. 3407ś3414.
  17. [88]88. Roger E Kirk. 2007. Experimental design. Wiley Online Library.
  18. [90]90. Patrick Koch, Brett Wujek, Oleg Golovidov, and Steven Gardner. 2017. Automated hyperparameter tuning for effective machine learning. In SAS Global Forum Conference.
  19. [92]92. Philip Koopman and Frank Fratrik. 2019. How Many Operational Design Domains, Objects, and Events?. In AAAI Workshop on Artificial Intelligence Safety.
  20. [93]93. Philip Koopman, Aaron Kane, and Jen Black. 2019. Credible autonomy safety argumentation. In 27th Safety-Critical Systems Symposium.
  21. [94]94. SB Kotsiantis, Dimitris Kanellopoulos, and PE Pintelas. 2006. Data preprocessing for supervised leaning. International Journal of Computer Science 1, 2 (2006), 111ś117.
  22. [97]97. Isaac Lage, Andrew Ross, Kim Been, Samuel Gershman, and Finale Doshi-Velez. 2018. Human-in-the-Loop Inter- pretability Prior. In Conference on Neural Information Processing Systems. 10180ś10189.
  23. [100]100. Zachary C Lipton. 2016. The mythos of model interpretability. arXiv:1606.03490 · arXiv
  24. [112]112. Markus Maurer, J Christian Gerdes, Barbara Lenz, and Hermann Winner. 2016. Autonomous driving: technical, legal and social aspects. Springer Nature.
  25. [113]113. Christopher Meyer and Jörg Schwenk. 2013. SoK: Lessons learned from SSL/TLS attacks. In International Workshop on Information Security Applications. Springer, 189ś209.
  26. [115]115. Tom M. Mitchell. 1997. Machine Learning. McGraw-Hill.
  27. [120]120. Pamela A Munro and Barbara G Kanki. 2003. An analysis of ASRS maintenance reports on the use of minimum equipment lists. In R. Jensen, 12th International Symposium on Aviation Psychology. Ohio State University, Dayton, OH.
  28. [127]127. Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated whitebox testing of deep learning systems. In 26th Symposium on Operating Systems Principles. ACM, 1ś18.
  29. [129]129. Michael J Pont and Royan HL Ong. 2002. Using watchdog timers to improve the reliability of single-processor embedded systems: Seven new patterns and a case study. In First Nordic Conference on Pattern Languages of Programs.
  30. [130]130. Lutz Prechelt. 1998. Early stopping-but when? In Neural Networks: Tricks of the trade. Springer, 55ś69.
  31. [133]133. J Provost Foster, Fawcett Tom, and Kohavi Ron. 1998. The case against accuracy estimation for comparing induction algorithms. In 15th International Conference on Machine Learning. 445ś453.
  32. [142]142. Andrew Slavin Ross and Finale Doshi-Velez. 2018. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In 32nd AAAI Conference on Artificial Intelligence.
  33. [146]146. Stuart J Russell and Peter Norvig. 2016. Artificial intelligence: a modern approach. Pearson Education Limited.
  34. [151]151. Lawrence K Saul and Sam T Roweis. 2003. Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of machine learning research 4, Jun (2003), 119ś155.
  35. [155]155. Daniel Selsam, Percy Liang, and David L Dill. 2017. Developing bug-free machine learning systems with formal mathematics. In 34th International Conference on Machine Learning-Volume 70. JMLR. org, 3047ś3056.
  36. [156]156. Victor S Sheng and Jing Zhang. 2019. Machine learning with crowdsourcing: A brief summary of the past research and future directions. In AAAI Conference on Artificial Intelligence, Vol. 33. 9837ś9843.
  37. [157]157. Andy Shih, Arthur Choi, and Adnan Darwiche. 2018. Formal verification of Bayesian network classifiers. In Interna- tional Conference on Probabilistic Graphical Models. 427ś438.
  38. [160]160. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 1 (2014), 1929ś1958.
  39. [168]168. Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated testing of deep-neural-network- driven autonomous cars. In 40th International Conference on Software Engineering. ACM, 303ś314.
  40. [169]169. John Törnblom and Simin Nadjm-Tehrani. 2018. Formal verification of random forests in safety-critical applications. In International Workshop on Formal Techniques for Safety-Critical Systems. Springer, 55ś71.
  41. [170]170. Hoang-Dung Tran, Stanley Bak, Weiming Xiang, and Taylor T Johnson. 2020. Verification of Deep Convolutional Neural Networks Using ImageStars. arXiv:2004.05511 · arXiv
  42. [171]171. Hoang-Dung Tran, Xiaodong Yang, Diego Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, and Taylor T Johnson. 2020. NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems. arXiv:2004.05519 · arXiv
  43. [176]176. Kiri L Wagstaff and Benjamin Bornstein. 2009. K-means in space: A radiation sensitivity evaluation. In 26th Annual International Conference on Machine Learning. 1097ś1104.
  44. [177]177. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In International Conference on Machine Learning. 1058ś1066.
  45. [186]186. Sebastien C Wong, Adam Gatt, Victor Stamatescu, and Mark D McDonnell. 2016. Understanding data augmentation for classification: when to warp?. In International Conference on digital image computing: techniques and applications. IEEE, 1ś6.
  46. [187]187. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V., Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.0814

Authors
Ashmore, R., Calinescu, R., & Paterson, C.
Affiliations
  • Defence Science and Technology Laboratory (GB)
  • University of York and Assuring Autonomy International Programme, UK
  • University of York (GB)
ORCID
Paterson, Colin
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2021-05-25
Volume
54
Issue
5
Pages
1-39
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3453444
OpenAlex
W3165722314
Semantic Scholar
8b7a185629d5480d207b2d3d74790b9b2ab17015
Cited by
256
References
122
Citations by year
2026: 55 · 2025: 54 · 2024: 51 · 2023: 57 · 2022: 29 · 2021: 9 · 2019: 1
FWCI
12.79
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Computing Surveys (publishedVersion)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
Funders
Assuring Autonomy International Programme and the UKRI project (EP/V026747/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
machine learning lifecycle, safety-critical systems, safety evidence, open problems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review