Towards Regulatory-Compliant MLOps: Oravizio’s Journey from a Machine Learning Experiment to a Deployed Certified Medical Product
Open accessAbstract
Agile software development embraces change and manifests working software over comprehensive documentation and responding to change over following a plan. The ability to continuously release software has enabled a development approach where experimental features are put to use, and, if they stand the test of real use, they remain in production. Examples of such features include machine learning (ML) models, which are usually pre-trained, but can still evolve in production. However, many domains require more plan-driven approach to avoid hazard to environment and humans, and to mitigate risks in the process. In this paper, we start by presenting continuous software engineering practices in a regulated context, and then apply the results to the emerging practice of MLOps, or continuous delivery of ML features. Furthermore, as a practical contribution, we present a case study regarding Oravizio, first CE-certified medical software for assessing the risks of joint replacement surgeries. Towards the end of the paper, we also reflect the Oravizio experiences to MLOps in regulatory context.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Models packaged in containers are simply run directly as standalone services.
“Models packaged in containers are simply run directly as standalone services [16,19,28,29,45].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [19]
The model may be integrated directly into the application code, traditionally first having to be rewritten for production.
“The model may be integrated directly into the application code [19,29,45], traditionally first having to be rewritten for production [21].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Manifesto for agile software development
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., & Thomas, D. A. (2001) · Cited by 4,284 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148 · see citing works
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · see citing works
Embracing change with extreme programming
Beck, K. (1999) · Computer · doi:10.1109/2.796139 · Cited by 981 · see citing works
(2015) · neural information processing systems · Cited by 912 · see citing works
Roberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 902 · see citing works
Continuous software engineering: A roadmap and agenda
Fitzgerald, B. F., & Stol, K.-J. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 742 · see citing works
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Comprehensible classification models
Freitas, A. A. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2594473.2594475 · Cited by 575 · see citing works
Devops: A Software Architect's Perspective
Bass, L., Weber, I. M., & Zhu, L. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 561 · see citing works
Interpretable Machine Learning
Molnar, C. (2020) · Cited by 506 · see citing works
Using risk to balance agile and plan- driven methods
BOEHM, B. W., & Turner, R. (2003) · Computer · doi:10.1109/mc.2003.1204376 · Cited by 415 · see citing works
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350 · see citing works
Continuous software engineering and beyond: trends and challenges
Fitzgerald, B., & Stol, K.-J. (2014) · Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering · doi:10.1145/2593812.2593813 · Cited by 212 · see citing works
On testing machine learning programs
Braiek, H. B., & Khomh, F. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110542 · Cited by 203 · see citing works
Algorithms on regulatory lockdown in medicine
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2019) · Science · doi:10.1126/science.aay9547 · Cited by 123 · see citing works
Lwakatare, L. E., Kuvaja, P., & Oivo, M. (2015) · Lecture notes in business information processing · doi:10.1007/978-3-319-18612-2_19 · Cited by 121 · see citing works
McHugh, M., Cawley, O., McCaffcry, F., Richardson, I., & Wang, X. (2013) · Software Engineering in Health Care · doi:10.1109/sehc.2013.6602471 · Cited by 71 · see citing works
DevOps in Regulated Software Development: Case Medical Devices
Laukkarinen, T., Kuusinen, K., & Mikkonen, T. (2017) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2017.20 · Cited by 57 · see citing works
Self-citationDevOps culture and its impact on cloud delivery and software development
Rajkumar, M., Pole, A. K., Adige, V. S., et al. (2016) · International Conference on Advances in Computing, Communication & Automation (ICACCA) · doi:10.1109/icacca.2016.7578902 · Cited by 53 · see citing works
Marnau, N. (2019) · Figshare · doi:10.60882/cispa.24612372 · Cited by 47 · see citing works
Regulated software meets DevOps
Laukkarinen, T., Kuusinen, K., & Mikkonen, T. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.01.011 · Cited by 45 · see citing works
Self-citationMc Hugh, M., Cawley, O., Mc Caffery, F., Richardson, I., & Wang, X. (2013) · STOR (Dundalk Institute of Technology) · Cited by 42 · see citing works
Demystifying Data Science Projects: A Look on the People and Process of Data Science Today
Aho, T., Sievi-Korte, O., Kilamo, T., Yaman, S. G., & Mikkonen, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64148-1_10 · Cited by 40 · see citing works
Self-citationFeng, J., Emerson, S., & Simon, N. (2021) · Biometrics · doi:10.1111/biom.13379 · Cited by 26 · see citing works
Is Machine Learning Software Just Software: A Maintainability View
Mikkonen, T., Nurminen, J. K., Raatikainen, M., Fronza, I., Mäkitalo, N., & Männistö, T. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-65854-0_8 · Cited by 21 · see citing works
Self-citationDevOps in an ISO 13485 Regulated Environment: A Multivocal Literature Review
Lie, M. F., Sánchez-Gordón, M., & Colomo-Palacios, R. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410679 · Cited by 21 · see citing works
Fursin, G., Guillou, H., & Essayan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07935 · Cited by 15 · see citing works
On Medical Device Software CE Compliance and Conformity Assessment
Granlund, T., Mikkonen, T., & Stirbu, V. (2020) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c50368.2020.00040 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Debois P. Devops: a software revolution in the making. Cutter IT J. 2011;24(8):3-9.
- European Commission. Commission notice -the 'blue guide' on the implementation of eu products rules 2016. In: Official Journal of the European Union, volume 59, 2016.
Cited by
Sorted by citations
External validation of AI models in health should be replaced with recurring local validation
Youssef, A., Pencina, M., Thakur, A., Zhu, T., Clifton, D., & Shah, N. H. (2023) · Nature Medicine · doi:10.1038/s41591-023-02540-z · Cited by 170 · see citing works
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
- Authors
- Granlund, T., Stirbu, V., & Mikkonen, T.
- Affiliations
- University of Helsinki (FI)
- Corresponding author
- Granlund, Tuomas
- Venue
- SN Computer Science
- Published
- 2021-06-19
- Volume
- 2
- Issue
- 5
- ISSN
- 2661-8907 (ISSN-L)
- 2662-995X
- OpenAlex
- W3173148850
- Cited by
- 53
- References
- 31
- Citations by year
- 2026: 7 · 2025: 11 · 2024: 12 · 2023: 14 · 2022: 8 · 2021: 1
- FWCI
- 7.91
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- SN Computer Science (publishedVersion, cc-by)
- Työväentutkimus Vuosikirja (submittedVersion)
- Funders
- Business Finland
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- MLOps, regulatory compliance, medical device regulation, CE certification, continuous software engineering, agile software development, risk mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
