A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation
Open accessAbstract
Artificial intelligence enabled systems have been an inevitable part of everyday life. However, efficient software engineering principles and processes need to be considered and extended when developing AI- enabled systems. The objective of this study is to identify and classify software engineering challenges that are faced by different companies when developing software-intensive systems that incorporate machine learning components. Using case study approach, we explored the development of machine learning systems from six different companies across various domains and identified main software engineering challenges. The challenges are mapped into a proposed taxonomy that depicts the evolution of use of ML components in software-intensive system in industrial settings. Our study provides insights to software engineering community and research to guide discussions and future research into applied machine learning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Serving-related challenges are some of the ones most often reported in the literature, usually from a performance perspective.
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models.
“Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models [6,7,12,28,34].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862 · see citing works
(2015) · neural information processing systems · Cited by 912 · see citing works
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590 · see citing works
Guest Editors' Introduction: On Applied Research in Machine Learning
Provost, F. J., & Kohavi, R. (1998) · Machine Learning · doi:10.1023/a:1007442505281 · Cited by 377 · see citing works
Large-scale machine learning at twitter
Lin, J., & Kolcz, A. (2012) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2213836.2213958 · Cited by 221 · see citing works
Software Engineering Challenges of Deep Learning
Arpteg, A., Brinne, B., Crnkovic-Friis, L., & Bosch, J. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00018 · Cited by 219 · see citing works
Self-citationData Management Challenges in Production Machine Learning
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3054782 · Cited by 214 · see citing works
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184 · see citing works
Cascade Ranking for Operational E-commerce Search
Liu, S., Xiao, F., Ou, W., & Si, L. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098011 · Cited by 111 · see citing works
Trials and tribulations of developers of intelligent systems: A field study
Hill, C., Bellamy, R., Erickson, T. D., & Burnett, M. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739680 · Cited by 108 · see citing works
Automatically Tracking Metadata and Provenance of Machine Learning Experiments
Schelter, S., Böse, J.-H., Kirschnick, J., Klein, T., & Seufert, S. (2017) · Cited by 78 · see citing works
Attenberg, J., & Provost, F. J. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964906 · Cited by 77 · see citing works
An Approach to Software Testing of Machine Learning Applications
Murphy, C., Kaiser, G. E., & Arias, M. (2007) · doi:10.7916/d8r49znr · Cited by 60 · see citing works
Tata, S., Popescul, A., Najork, M., Colagrosso, M., Gibbons, J., Green, A. I., Mah, A., Smith, M. D., Garg, D., Meyer, C., et al. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098048 · Cited by 46 · see citing works
Practical Machine Learning for Cloud Intrusion Detection
Kumar, R. S. S., Wicker, A., & Swann, M. (2017) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3128572.3140445 · Cited by 43 · see citing works
It takes three to tango: Requirement, outcome/data, and AI driven development
Bosch, J., Olsson, H. H., & Crnković, I. (2018) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 38 · see citing works
Self-citationA Framework for Adoption of Machine Learning in Industry for Software Defect Prediction
Rana, R., Staron, M., Hansson, J., Nilsson, M., & Meding, W. (2014) · International Conference on Software Engineering and Applications (ICSOFT-EA) · doi:10.5220/0005099303830392 · Cited by 37 · see citing works
Design principles of massive, robust prediction systems
Raeder, T., Stitelman, O., Dalessandro, B., Perlich, C., & Provost, F. J. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339740 · Cited by 30 · see citing works
Versioning for End-to-End Machine Learning Pipelines
van der Weide, T., Papadopoulos, D., Smirnov, O., Zielinski, M., & van Kasteren, T. L. M. (2017) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3076246.3076248 · Cited by 28 · see citing works
Hains, G., Jakobsson, A., & Khmelevsky, Y. (2018) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon.2018.8369576 · Cited by 28 · see citing works
An Architecture for Agile Machine Learning in Real-Time Applications
Schleier-Smith, J. (2015) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2788628 · Cited by 21 · see citing works
Sjafrie, H. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429316777-2 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Lefortier, D., Truchet, A., de Rijke, M.: Sources of variability in large-scale machine learning systems. In: Machine Learning Systems (NIPS 2015 Workshop) (2015)
Cited by
Sorted by citations
Perifanis, N.-A., & Kitsios, F. (2023) · Information · doi:10.3390/info14020085 · Cited by 407 · see citing works
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 374 · see citing works
Taxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380395 · Cited by 299 · see citing works
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241 · see citing works
Self-citationGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Collaboration challenges in building ML-enabled systems
Nahar, N., Zhou, S., Lewis, G. A., & Kästner, C. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510209 · Cited by 164 · see citing works
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151 · see citing works
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132 · see citing works
Visual tools for teaching machine learning in K-12: A ten-year systematic mapping
Gresse von Wangenheim, C., Hauck, J. C. R., Pacheco, F. S., & Bertonceli Bueno, M. F. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10570-8 · Cited by 130 · see citing works
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127 · see citing works
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126 · see citing works
Requirements Engineering Challenges in Building AI-Based Complex Systems
Belani, H., Vuković, M., & Car, Ž. (2019) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew.2019.00051 · Cited by 113 · see citing works
Architectural patterns for the design of federated learning systems
Lo, S. K., Lu, Q., Zhu, L., Paik, H.-Y., Xu, X., & Wang, C. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111357 · Cited by 109 · see citing works
Engineering AI Systems: A Research Agenda
Bosch, J., Olsson, H. H., & Crnkovic, I. (2021) · Advances in Systems Analysis, Software Engineering, and High Performance Computing · doi:10.4018/978-1-7998-5101-1.ch001 · Cited by 108 · see citing works
Self-citationThe Current State of Industrial Practice in Artificial Intelligence Ethics
Vakkuri, V., Kemell, K., Kultanen, J., & Abrahamsson, P. (2020) · IEEE Software · doi:10.1109/ms.2020.2985621 · Cited by 98 · see citing works
End-to-End Federated Learning for Autonomous Driving Vehicles
Zhang, H., Bosch, J., & Olsson, H. H. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533808 · Cited by 84 · see citing works
Self-citationA Literature Review of Using Machine Learning in Software Development Life Cycle Stages
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2021) · IEEE Access · doi:10.1109/access.2021.3119746 · Cited by 79 · see citing works
Gonzalez, D., Zimmermann, T., & Nagappan, N. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387473 · Cited by 77 · see citing works
Software Architecture for ML-based Systems: What Exists and What Lies Ahead
Muccini, H., & Vaidhyanathan, K. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00026 · Cited by 73 · see citing works
Nascimento, E., Nguyen‐Duc, A., Sundbø, I., & Conte, T. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03751 · Cited by 69 · see citing works
An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems
Tang, Y., Khatchadourian, R., Bagherzadeh, M., Singh, R., Stewart, A., & Raja, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00033 · Cited by 68 · see citing works
Requirements engineering framework for human-centered artificial intelligence software systems
Ahmad, K., Abdelrazek, M., Arora, C., Baniya, A. A., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110455 · Cited by 65 · see citing works
Engineering AI Systems: A Research AgendaPreprint
Bosch, J., Crnković, I., & Olsson, H. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07522 · Cited by 64 · see citing works
Self-citationHabibullah, K. M., Gay, G., & Horkoff, J. (2023) · Requirements Engineering · doi:10.1007/s00766-022-00395-3 · Cited by 63 · see citing works
What’s up with Requirements Engineering for Artificial Intelligence Systems?
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., & Grundy, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00008 · Cited by 61 · see citing works
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 58 · see citing works
Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
Windmann, A., Wittenberg, P., Schieseck, M., & Niggemann, O. (2024) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin58382.2024.10774364 · Cited by 58 · see citing works
Software Architecture Challenges for ML Systems
Lewis, G. A., Özkaya, İ., & Xu, X. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00071 · Cited by 57 · see citing works
Subramonyam, H., Im, J., Seifert, C. M., & Adar, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517537 · Cited by 57 · see citing works
Towards Guidelines for Assessing Qualities of Machine Learning Systems
Siebert, J., Jöckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-58793-2_2 · Cited by 56 · see citing works
DevOps for AI – Challenges in Development of AI-enabled Applications
Lwakatare, L. E., Crnković, I., & Bosch, J. (2020) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom50211.2020.9238323 · Cited by 55 · see citing works
Self-citationCharacterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study
Bogner, J., Verdecchia, R., & Gerostathopoulos, I. (2021) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1109/techdebt52882.2021.00016 · Cited by 54 · see citing works
Real-time End-to-End Federated Learning: An Automotive Case Study
Zhang, H., Bosch, J., & Olsson, H. H. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00070 · Cited by 54 · see citing works
Self-citationTowards an AI‐driven business development framework: A multi‐case study
John, M. M., Olsson, H. H., & Bosch, J. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2432 · Cited by 53 · see citing works
Self-citationHabibullah, K. M., & Horkoff, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00009 · Cited by 5 · see citing works
Habiba, U.-E., Haug, M., Bogner, J., & Wagner, S. (2024) · Requirements Engineering · doi:10.1007/s00766-024-00432-3 · Cited by 51 · see citing works
Recupito, G., Pecorelli, F., Catolino, G., Lenarduzzi, V., Taibi, D., Nucci, D. D., & Palomba, F. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112151 · Cited by 47 · see citing works
Going digital: Disruption and transformation in software‐intensive embedded systems ecosystems
Olsson, H. H., & Bosch, J. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2249 · Cited by 45 · see citing works
Self-citationAdapting Software Architectures to Machine Learning Challenges
Serban, A., & Visser, J. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00029 · Cited by 43 · see citing works
Bosch, J., & Olsson, H. H. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2333 · Cited by 42 · see citing works
Self-citationFederated Learning Systems: Architecture Alternatives
Zhang, H., Bosch, J., & Olsson, H. H. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00047 · Cited by 41 · see citing works
Self-citationCharacterizing and Detecting Mismatch in Machine-Learning-Enabled Systems
Lewis, G. A., Bellomo, S., & Özkaya, İ. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00028 · Cited by 41 · see citing works
Gunda, S. K., Yettapu, S. D. R., Bodakunti, S., & Bikki, S. B. (2023) · International Journal of Artificial Intelligence Data Science and Machine Learning · doi:10.63282/3050-9262.ijaidsml-v4i1p112 · Cited by 40 · see citing works
Software Quality for AI: Where We Are Now?
Lenarduzzi, V., Lomio, F., Moreschini, S., Taibi, D., & Tamburri, D. A. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-65854-0_4 · Cited by 39 · see citing works
Model driven engineering for machine learning components: A systematic literature review
Naveed, H., Arora, C., Khalajzadeh, H., Grundy, J., & Haggag, O. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107423 · Cited by 38 · see citing works
von Wangenheim, C. G., Alves, N. D. C., Rauber, M. F., Hauck, J. C. R., & Yeter, I. H. (2021) · Informatics in Education · doi:10.15388/infedu.2022.18 · Cited by 38 · see citing works
Recent Advances in Data Engineering for Networking
Zeydan, E., & Mangues‐Bafalluy, J. (2022) · IEEE Access · doi:10.1109/access.2022.3162863 · Cited by 38 · see citing works
Architectural Design Decisions for Machine Learning Deployment
Warnett, S. J., & Zdun, U. (2022) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa53651.2022.00017 · Cited by 38 · see citing works
Multilayered review of safety approaches for machine learning-based systems in the days of AI
Dey, S., & Lee, S.-W. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110941 · Cited by 37 · see citing works
Rabe, M., Milz, S., & Mäder, P. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00023 · Cited by 36 · see citing works
Maintainability and Scalability in Machine Learning: Challenges and Solutions
Shivashankar, K., Hajj, G. S. A., & Martini, A. (2025) · ACM Computing Surveys · doi:10.1145/3736751 · Cited by 35 · see citing works
John, M. M., Olsson, H. H., & Bosch, J. (2020) · International Conference on Software and System Process · doi:10.1145/3379177.3388892 · Cited by 35 · see citing works
Self-citationOperationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
Benchmarking Machine Learning Solutions in Production
Cardoso Silva, L., Rezende Zagatti, F., Silva Sette, B., Nildaimon dos Santos Silva, L., Lucredio, D., Furtado Silva, D., & de Medeiros Caseli, H. (2020) · 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla51294.2020.00104 · Cited by 33 · see citing works
Trustworthy artificial intelligence: A decision‐making taxonomy of potential challenges
Akbar, M. A., Khan, A. A., Mahmood, S., Rafi, S., & Demi, S. (2023) · Software Practice and Experience · doi:10.1002/spe.3216 · Cited by 33 · see citing works
Identifying concerns when specifying machine learning-enabled systems: A perspective-based approach
Villamizar, H., Kalinowski, M., Lopes, H., & Méndez, D. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112053 · Cited by 33 · see citing works
Architectural Design Decisions for the Machine Learning Workflow
Warnett, S. J., & Zdun, U. (2022) · Computer · doi:10.1109/mc.2021.3134800 · Cited by 32 · see citing works
Dhanekula, A., & Miah, M. R. (2022) · American Journal of Interdisciplinary Studies · doi:10.63125/707veh84 · Cited by 30 · see citing works
Software engineering practices for machine learning — Adoption, effects, and team assessment
Serban, A., van der Blom, K., Hoos, H. H., & Visser, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111907 · Cited by 29 · see citing works
Requirements Engineering in Machine Learning Projects
Gjorgjevikj, A., Mishev, K., Antovski, L., & Trajanov, D. (2023) · IEEE Access · doi:10.1109/access.2023.3294840 · Cited by 29 · see citing works
A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks
Casey, B., Santos, J. C. S., & Perry, G. (2025) · ACM Computing Surveys · doi:10.1145/3721977 · Cited by 28 · see citing works
Emerging and Changing Tasks in the Development Process for Machine Learning Systems
Liu, H., Eksmo, S., Risberg, J., & Hebig, R. (2020) · International Conference on Software and System Process · doi:10.1145/3379177.3388905 · Cited by 24 · see citing works
Challenges in developing and deploying AI in the engineering, procurement and construction industry
Джусупова, Р., Bosch, J., & Olsson, H. H. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00167 · Cited by 22 · see citing works
Self-citationChapter 1 Climbing the Stairway to Heaven
Olsson, H. H., Alahyari, H., & Bosch, J. (2012) · Springer eBooks · doi:10.1007/978-3-031-10873-0_2 · Cited by 21 · see citing works
Self-citationRequirements and software engineering for automotive perception systems: an interview study
Habibullah, K. M., Heyn, H.-M., Gay, G., Horkoff, J., Knauss, E., Borg, M., Knauss, A., Sivencrona, H., & Li, P. J. (2024) · Requirements Engineering · doi:10.1007/s00766-023-00410-1 · Cited by 20 · see citing works
A Multiple Case Study of Artificial Intelligent System Development in Industry
Nguyen‐Duc, A., Sundbø, I., Nascimento, E., Conte, T. U., Ahmed, I., & Abrahamsson, P. (2020) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3383219.3383220 · Cited by 19 · see citing works
A Holistic Framework for AI Systems in Industrial Applications
Kaymakci, C., Wenninger, S., & Sauer, A. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_6 · Cited by 19 · see citing works
Practices for Managing Machine Learning Products: A Multivocal Literature Review
Alves, I., Leite, L., Meirelles, P., Kon, F., & Rocha, C. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3287759 · Cited by 18 · see citing works
R, D., & Mathew, J. (2025) · Advances in Science and Technology – Research Journal · doi:10.12913/22998624/197406 · Cited by 18 · see citing works
Effects of Artificial Intelligence on Enterprise Architectures - A Structured Literature Review
Rittelmeyer, J. D., & Sandkuhl, K. (2021) · IEEE International Enterprise Distributed Object Computing Conference Workshops (EDOCW) · doi:10.1109/edocw52865.2021.00042 · Cited by 18 · see citing works
An empirical study on ML DevOps adoption trends, efforts, and benefits analysis
Rzig, D. E., Hassan, F., & Kessentini, M. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107037 · Cited by 18 · see citing works
AI Engineering: Realizing the Potential of AI
Bosch, J., Olsson, H. H., Brinne, B., & Crnković, I. (2022) · IEEE Software · doi:10.1109/ms.2022.3199621 · Cited by 17 · see citing works
Self-citationLandscape of Requirements Engineering for Machine Learning-based AI Systems
Yoshioka, N., Husen, J. H., Tun, H. T., Chen, Z., Washizaki, H., & Fukazawa, Y. (2021) · doi:10.1109/apsecw53869.2021.00011 · Cited by 16 · see citing works
Common challenges of deep reinforcement learning applications development: an empirical study
Morovati, M. M., Tambon, F., Taraghi, M., Nikanjam, A., & Khomh, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10500-5 · Cited by 16 · see citing works
Enabling efficient and low-effort decentralized federated learning with the EdgeFL framework
Zhang, H., Bosch, J., & Olsson, H. H. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107600 · Cited by 16 · see citing works
Self-citationState-of-the-Art and Challenges of Engineering ML- Enabled Software Systems in the Deep Learning Era
Assres, G. M., Bhandari, G. P., Shalaginov, A., Grønli, T., & Ghinea, G. (2025) · ACM Computing Surveys · doi:10.1145/3731597 · Cited by 15 · see citing works
What Machine Learning Can Learn From Software Modularity
Kriens, P., & Verbelen, T. (2022) · Computer · doi:10.48550/arxiv.1906.10366 · Cited by 15 · see citing works
Preliminary Systematic Literature Review of Machine Learning System Development Process
Watanabe, Y., Washizaki, H., Sakamoto, K., Saito, D., Honda, K., Tsuda, N., Fukazawa, Y., & Yoshioka, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05528 · Cited by 15 · see citing works
Making Sense of AI Systems Development
Dolata, M., & Crowston, K. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3338857 · Cited by 15 · see citing works
Time for AI (Ethics) Maturity Model Is NowPreprint
Vakkuri, V., Jantunen, M., Halme, E., Kemell, K., Nguyen‐Duc, A., Mikkonen, T., & Abrahamsson, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12701 · Cited by 15 · see citing works
Research challenges for machine learning-constructed software
Pahl, C. (2022) · Service Oriented Computing and Applications · doi:10.1007/s11761-022-00352-6 · Cited by 15 · see citing works
Ibarra-Torres, F., Caiza, G. J., García, M. V., & Barona-Pico, V. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.05.122 · Cited by 14 · see citing works
Towards Human-Centric Psychomotor Recommender Systems
Portaz, M., Manjarrés, Á., & Santos, O. (2023) · User Modeling, Adaptation, and Personalization · doi:10.1145/3563359.3596993 · Cited by 10 · see citing works
Papagiannidis, E., Mikalef, P., Krogstie, J., & Conboy, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15342-6_5 · Cited by 13 · see citing works
Towards a Taxonomy of Cognitive RPA Components
Martínez-Rojas, A., Barba, I., & Enríquez, J. G. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-58779-6_11 · Cited by 12 · see citing works
Data sovereignty for AI pipelines
Altendeitering, M., Pampus, J., Larrinaga, F., Legaristi, J., & Howar, F. M. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528593 · Cited by 12 · see citing works
Human-Centered Recommender Systems
Portaz, M., Riesco, Á. M., & Santos, O. C. (2025) · Human-computer interaction series · doi:10.1007/978-3-031-61375-3_11 · Cited by 11 · see citing works
Parida, S. K., Gerostathopoulos, I., & Bogner, J. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00014 · Cited by 11 · see citing works
The Product Beyond the Model - An Empirical Study of Repositories of Open-Source ML Products
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00006 · Cited by 11 · see citing works
Rauber, M. F., von Wangenheim, C. G., Barbetta, P. A., Borgatto, A. F., Martins, R. M., & Hauck, J. C. R. (2023) · Informatics in Education · doi:10.15388/infedu.2024.10 · Cited by 11 · see citing works
Hidden Feedback Loops in Machine Learning Systems: A Simulation Model and Preliminary Results
Khritankov, A. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-65854-0_5 · Cited by 11 · see citing works
AI Deployment Architecture: Multi-Case Study for Key Factor Identification
John, M. M., Olsson, H. H., & Bosch, J. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00048 · Cited by 11 · see citing works
Self-citationJentzsch, S. F., & Hochgeschwender, N. (2019) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew.2019.00025 · Cited by 11 · see citing works
Machine Learning for Software Engineering: A Systematic MappingPreprint
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13299 · Cited by 11 · see citing works
Anomaly analytics in data-driven machine learning applications
Azimi, S., & Pahl, C. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00593-y · Cited by 10 · see citing works
EdgeFL: A Lightweight Decentralized Federated Learning Framework
Zhang, H., Bosch, J., & Olsson, H. H. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00081 · Cited by 10 · see citing works
Self-citationInterplay of Machine Learning and Software Engineering for Quality Estimations
Abubakar, H., Obaidat, M. S., Gupta, A., Bhattacharya, P., & Tanwar, S. (2020) · International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI) · doi:10.1109/ccci49893.2020.9256507 · Cited by 10 · see citing works
AF-DNDF: Asynchronous Federated Learning of Deep Neural Decision Forests
Zhang, H., Bosch, J., Olsson, H. H., & Koppisetty, A. C. (2021) · 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00047 · Cited by 10 · see citing works
Self-citationNaming the Pain in machine learning-enabled systems engineering
Kalinowski, M., Méndez, D., Giray, G., Alves, A. P. S., Azevedo, K., Escovedo, T., Villamizar, H., Lopes, H., Baldassarre, M. T., Wagner, S., et al. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107866 · Cited by 10 · see citing works
Continuous experimentation on artificial intelligence software: a research agenda
Nguyen‐Duc, A., & Abrahamsson, P. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417039 · Cited by 10 · see citing works
- Authors
- Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., & Crnkovic, I.
- Affiliations
- Chalmers University of Technology (SE)
- Malmö University (SE)
- Corresponding author
- Lwakatare, Lucy Ellen
- Venue
- Lecture Notes in Business Information Processing
- Publisher
- Springer International Publishing
- Published
- 2019
- Pages
- 227-243
- ISSN
- 1865-1348 (ISSN-L)
- 1865-1356
- OpenAlex
- W2944264045
- Cited by
- 214
- References
- 22
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 27 · 2023: 31 · 2022: 43 · 2021: 44 · 2020: 29 · 2019: 10 · 2018: 1 · 2017: 1 · 2015: 2 · 2013: 1 · 2012: 1
- FWCI
- 25.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Lecture notes in business information processing (publishedVersion, cc-by)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Malmö University Publications (Malmö University) (submittedVersion)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
- Keywords
- software engineering challenges, machine learning systems, artificial intelligence systems, software-intensive systems, industrial software development, applied machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
