Systematic Mapping Study on the Machine Learning Lifecycle
Open accessAbstract
The development of artificial intelligence (AI) has made various industries eager to explore the benefits of AI. There is an increasing amount of research surrounding AI, most of which is centred on the development of new AI algorithms and techniques. However, the advent of AI is bringing an increasing set of practical problems related to AI model lifecycle management that need to be investigated. We address this gap by conducting a systematic mapping study on the lifecycle of AI model. Through quantitative research, we provide an overview of the field, identify research opportunities, and provide suggestions for future research. Our study yields 405 publications published from 2005 to 2020, mapped in 5 different main research topics, and 31 sub-topics. We observe that only a minority of publications focus on data management and model production problems, and that more studies should address the AI lifecycle from a holistic perspective.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
[52] found that AI model deployment was underexplored.
“[52] was a SMS on the lifecycle of the AI models and found that AI model deployment was underexplored.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [52]
Cites
Systematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · see citing works
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669 · see citing works
Requirements engineering paper classification and evaluation criteria: a proposal and a discussion
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2006) · Requirements Engineering · doi:10.1007/s00766-005-0021-6 · Cited by 940 · see citing works
Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0
Bourque, P., & Fairley, R. E. (2014) · IEEE Computer Society Press eBooks · Cited by 455 · see citing works
Barricelli, B. R., Cassano, F., Fogli, D., & Piccinno, A. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.11.041 · Cited by 269 · see citing works
Software Engineering Challenges of Deep Learning
Arpteg, A., Brinne, B., Crnkovic-Friis, L., & Bosch, J. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00018 · Cited by 219 · see citing works
On the reliability of mapping studies in software engineering
Wohlin, C., Runeson, P., Neto, P. A. D. M. S., Engström, E., Machado, I., & de Almeida, E. S. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.04.076 · Cited by 207 · see citing works
Artificial Intelligence for Cybersecurity: A Systematic Mapping of Literature
Wiafe, I., Koranteng, F. N., Obeng, E. N., Assyne, N., Wiafe, A., & Gulliver, S. R. (2020) · IEEE Access · doi:10.1109/access.2020.3013145 · Cited by 157 · see citing works
AI lifecycle models need to be revised
Haakman, M., Cruz, L., Huijgens, H., & Arie Van Deursen, A. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09993-1 · Cited by 125 · see citing works
Self-citationSoftware engineering for machine-learning applications: The road ahead
Khomh, F., Adams, B., Cheng, J., Fokaefs, M., & Antoniol, G. (2018) · IEEE Software · doi:10.1109/ms.2018.3571224 · Cited by 95 · see citing works
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81 · see citing works
Nascimento, E., Nguyen‐Duc, A., Sundbø, I., & Conte, T. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03751 · Cited by 69 · see citing works
Benefitting from the Grey Literature in Software Engineering Research
Garousi, V., Felderer, M., Mäntylä, M., & Rainer, A. (2020) · Springer eBooks · doi:10.1007/978-3-030-32489-6_14 · Cited by 62 · see citing works
ModelHub: Deep Learning Lifecycle Management
Miao, H., Li, A., Davis, L. S., & Deshpande, A. (2017) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2017.192 · Cited by 61 · see citing works
The Key Concepts of Ethics of Artificial Intelligence
Vakkuri, V., & Abrahamsson, P. (2018) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice.2018.8436265 · Cited by 52 · see citing works
AI Lifecycle Models Need To Be Revised. An Exploratory Study in FintechPreprint
Haakman, M., Cruz, L., Huijgens, H., & Arie Van Deursen, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02716 · Cited by 16 · see citing works
Self-citationA Systematic Mapping Study on Testing of Machine Learning ProgramsPreprint
Sherin, S., Khan, M. U., & Iqbal, M. Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09427 · Cited by 8 · see citing works
Systematic literature reviews: A Case Study in FinTech and Automated Tool Support
Spaargaren, W. (2020) · Research Repository (Delft University of Technology) · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126 · see citing works
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
- Authors
- Xie, Y., Cruz, L., Heck, P., & Rellermeyer, J. S.
- Affiliations
- ING Bank (NL)
- Fontys University of Applied Sciences (NL)
- Delft University of Technology (NL)
- Venue
- 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN)
- Publisher
- IEEE
- Published
- 2021-05
- Pages
- 70-73
- OpenAlex
- W3178745199
- Cited by
- 14
- References
- 18
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 6 · 2023: 2 · 2022: 3
- FWCI
- 1.27
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Research Repository (Delft University of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- machine learning lifecycle, AI model deployment, data management, research gaps, research opportunities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
