CoChat
Sign Up
PreprintCited by 22Safety Systems Engineering in AutonomySoftware Engineering Techniques and PracticesAdvanced Software Engineering Methodologies

The DOI matches its DataCite record.

Developing and Operating Artificial Intelligence Models in Trustworthy Autonomous Systems

Open access
Martínez-Fernández, Silverio, Franch, Xavier, Jedlitschka, Andreas, Oriol, Marc, Trendowicz, Adam·2020·arXiv
DOI 10.48550/arxiv.2003.05434arXiv 2003.05434
Open this paper in CoChat — free accountOpen PDF

Abstract

Companies dealing with Artificial Intelligence (AI) models in Autonomous Systems (AS) face several problems, such as users' lack of trust in adverse or unknown conditions, gaps between software engineering and AI model development, and operation in a continuously changing operational environment. This work-in-progress paper aims to close the gap between the development and operation of trustworthy AI-based AS by defining an approach that coordinates both activities. We synthesize the main challenges of AI-based AS in industrial settings. We reflect on the research efforts required to overcome these challenges and propose a novel, holistic DevOps approach to put it into practice. We elaborate on four research directions: (a) increased users' trust by monitoring operational AI-based AS and identifying self-adaptation needs in critical situations; (b) integrated agile process for the development and evolution of AI models and AS; (c) continuous deployment of different context-specific instances of AI models in a distributed setting of AS; and (d) holistic DevOps-based lifecycle for AI-based AS.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2003.05434

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 23

  1. [1]1. M. Arnold, R. K. Bellamy et al. , 'FactSheets: Increasing trust in AI services through supplier's declarations of conformity,' IBM Journal of Research and Development , vol. 63, no. 4/5, pp. 6-1, 2019.
  2. [2]2. M. Gharib, P. Lollini et al. , 'On the Safety of Automotive Systems Incorporating Machine Learning Based Components,' in DSN-W , 2018, pp. 271-274.
  3. [3]3. M. Borg, C. Englund et al. , 'Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry,' Journal of Automotive Software Engineering , vol. 1, no. 1, p. 1, 2019.
  4. [4]4. I. Ozkaya, 'What Is Really Different in Engineering AI-Enabled Systems?' IEEE Software , vol. 37, no. 4, pp. 3-6, 2020.
  5. [5]5. A. Serban, K. van der Blom, H. Hoos, and J. Visser, 'Adoption and Effects of Software Engineering Best Practices in Machine Learning,' arXiv:2007.14130 , 2020. · arXiv
  6. [6]6. A. Aniculaesei, J. Grieser et al. , 'Toward a holistic software systems engineering approach for dependable autonomous systems,' in SEFAIAS , 2018, pp. 23-30.
  7. [8]8. M. T. Ribeiro, S. Singh, and C. Guestrin, 'Why should I trust you? Explaining the predictions of any classifier,' in ACM SIGKDD , 2016, pp. 1135-1144.
  8. [9]9. R. Akkiraju, V. Sinha et al. , 'Characterizing machine learning processes: A maturity framework,' in BPM . Springer, 2020, pp. 17-31.
  9. [10]10. M. Kim, T. Zimmermann, R. DeLine, and A. Begel, 'Data scientists in software teams: State of the art and challenges,' TSE , vol. 44, no. 11, pp. 1024-1038, 2017.
  10. [12]12. P. Santhanam, E. Farchi, and V. Pankratius, 'Engineering Reliable Deep Learning Systems,' arXiv:1910.12582 , 2019. · arXiv
  11. [14]14. L. E. Lwakatare, A. Raj, I. Crnkovic, J. Bosch, and H. H. Olsson, 'Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions,' Information and Software Technology , vol. 127, p. 106368, 2020.
  12. [15]15. R. Garcia, V. Sreekanti et al. , 'Context: The missing piece in the machine learning lifecycle,' in KDD CMI Workshop , vol. 114, 2018.
  13. [16]16. A. Nguyen-Duc, I. Sundbø et al. , 'A Multiple Case Study of Artificial Intelligent System Development in Industry,' in EASE , 2020, pp. 1-10.
  14. [17]17. C. K¨ astner and E. Kang, 'Teaching Software Engineering for AI-Enabled Systems,' arXiv preprint arXiv:2001.06691 , 2020. · arXiv
  15. [18]18. A. Banijamali, P. Jamshidi, P. Kuvaja, and M. Oivo, 'Kuksa: A Cloud-Native Architecture for Enabling Continuous Delivery in the Automotive Domain,' in PROFES , 2019, pp. 455-472.
  16. [19]19. IBM, 'An architectural blueprint for autonomic computing,' IBM White Paper , vol. 31, no. 2003, pp. 1-6, 2006.
  17. [20]20. J. Siebert, L. Joeckel, J. Heidrich, K. Nakamichi, K. Ohashi, I. Namba, R. Yamamoto, and M. Aoyama, 'Towards guidelines for assessing qualities of machine learning systems,' in QUATIC . Springer, 2020, pp. 17-31.
  18. [23]23. E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, 'The ML test score: A rubric for ML production readiness and technical debt reduction,' in Big Data , 2017, pp. 1123-1132.
  19. [25]25. T. Gustavsson, 'Benefits of Agile Project Management in a Non-Software Development Context: A Literature Review,' in PM World Journal , 2016, pp. 114-124.
  20. [26]26. C. Shearer, 'The CRISP-DM model: the new blueprint for data mining,' Journal of data warehousing , vol. 5, no. 4, pp. 13-22, 2000.
  21. [27]27. S. Studer, T. B. Bui et al. , 'Towards CRISP-ML (Q): A Machine Learning Process Model with Quality Assurance Methodology,' arXiv:2003.05155 , 2020. · arXiv
  22. [31]31. L. Torrey and J. Shavlik, 'Transfer learning,' in Handbook of research on machine learning applications and trends . IGI global, 2010, pp. 242-264.
  23. [35]35. C. Ebert, G. Gallardo, J. Hernantes, and N. Serrano, 'DevOps,' IEEE Software , vol. 33, no. 3, pp. 94-100, 2016.

Cited by

Sorted by citations

Metadata

Authors
Martínez-Fernández, S., Franch, X., Jedlitschka, A., Oriol, M., & Trendowicz, A.
Affiliations
  • Universitat Politècnica de Catalunya (ES)
  • Fraunhofer Institute for Experimental Software Engineering (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
221-229
ISSN
1865-1348 (ISSN-L)
DOI
10.48550/arxiv.2003.05434
arXiv
2003.05434
OpenAlex
W3158418101
Cited by
22
References
0
Citations by year
2026: 4 · 2025: 3 · 2024: 3 · 2023: 7 · 2022: 4 · 2021: 1
FWCI
9.21
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Lecture notes in business information processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UPCommons institutional repository (Universitat Politècnica de Catalunya) (submittedVersion, other-oa)
  • Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • arXiv (Cornell University)
Funders
Agència de Gestió d'Ajuts Universitaris i de Recerca (2017SGR)
Topics
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
trustworthy AI, autonomous systems, AI model development, DevOps, AI lifecycle management, user trust, AI monitoring, self-adaptation, agile software development, continuous deployment, distributed artificial intelligence, industrial AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review