The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation
Open accessAbstract
Artificial intelligence enabled systems have been an inevitable part of everyday life. However, efficient software engineering principles and processes need to be considered and extended when developing AI- enabled systems. The objective of this study is to identify and classify software engineering challenges that are faced by different companies when developing software-intensive systems that incorporate machine learning components. Using case study approach, we explored the development of machine learning systems from six different companies across various domains and identified main software engineering challenges. The challenges are mapped into a proposed taxonomy that depicts the evolution of use of ML components in software-intensive system in industrial settings. Our study provides insights to software engineering community and research to guide discussions and future research into applied machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
In the citing paper
“Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models [6,7,12,28,34].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862
(2015) · neural information processing systems · Cited by 912
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590
Guest Editors' Introduction: On Applied Research in Machine Learning
Provost, F. J., & Kohavi, R. (1998) · Machine Learning · doi:10.1023/a:1007442505281 · Cited by 385
Software Engineering Challenges of Deep Learning
Arpteg, A., Brinne, B., Crnkovic-Friis, L., & Bosch, J. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00018 · Cited by 238
Self-citationLarge-scale machine learning at twitter
Lin, J., & Kolcz, A. (2012) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2213836.2213958 · Cited by 231
Data Management Challenges in Production Machine Learning
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3054782 · Cited by 214
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184
Trials and tribulations of developers of intelligent systems: A field study
Hill, C., Bellamy, R., Erickson, T. D., & Burnett, M. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739680 · Cited by 115
Cascade Ranking for Operational E-commerce Search
Liu, S., Xiao, F., Ou, W., & Si, L. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098011 · Cited by 111
Research works printed but not linked: 1
- Lefortier, D., Truchet, A., de Rijke, M.: Sources of variability in large-scale machine learning systems. In: Machine Learning Systems (NIPS 2015 Workshop) (2015)
Cited by
Sorted by citations
Perifanis, N.-A., & Kitsios, F. (2023) · Information · doi:10.3390/info14020085 · Cited by 407
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 426
Taxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380395 · Cited by 299
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Self-citationGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Collaboration challenges in building ML-enabled systems
Nahar, N., Zhou, S., Lewis, G. A., & Kästner, C. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510209 · Cited by 197
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Visual tools for teaching machine learning in K-12: A ten-year systematic mapping
Gresse von Wangenheim, C., Hauck, J. C. R., Pacheco, F. S., & Bertonceli Bueno, M. F. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10570-8 · Cited by 130
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
Requirements Engineering Challenges in Building AI-Based Complex Systems
Belani, H., Vuković, M., & Car, Ž. (2019) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew.2019.00051 · Cited by 113
Architectural patterns for the design of federated learning systems
Lo, S. K., Lu, Q., Zhu, L., Paik, H.-Y., Xu, X., & Wang, C. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111357 · Cited by 109
Engineering AI Systems: A Research Agenda
Bosch, J., Olsson, H. H., & Crnkovic, I. (2021) · Advances in Systems Analysis, Software Engineering, and High Performance Computing · doi:10.4018/978-1-7998-5101-1.ch001 · Cited by 108
Self-citationThe Current State of Industrial Practice in Artificial Intelligence Ethics
Vakkuri, V., Kemell, K., Kultanen, J., & Abrahamsson, P. (2020) · IEEE Software · doi:10.1109/ms.2020.2985621 · Cited by 98
End-to-End Federated Learning for Autonomous Driving Vehicles
Zhang, H., Bosch, J., & Olsson, H. H. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533808 · Cited by 84
Self-citationA Literature Review of Using Machine Learning in Software Development Life Cycle Stages
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2021) · IEEE Access · doi:10.1109/access.2021.3119746 · Cited by 79
Gonzalez, D., Zimmermann, T., & Nagappan, N. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387473 · Cited by 85
Software Architecture for ML-based Systems: What Exists and What Lies Ahead
Muccini, H., & Vaidhyanathan, K. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00026 · Cited by 73
Nascimento, E., Nguyen‐Duc, A., Sundbø, I., & Conte, T. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03751 · Cited by 78
An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems
Tang, Y., Khatchadourian, R., Bagherzadeh, M., Singh, R., Stewart, A., & Raja, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00033 · Cited by 68
Requirements engineering framework for human-centered artificial intelligence software systems
Ahmad, K., Abdelrazek, M., Arora, C., Baniya, A. A., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110455 · Cited by 70
Engineering AI Systems: A Research AgendaPreprint
Bosch, J., Crnković, I., & Olsson, H. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07522 · Cited by 68
Self-citationHabibullah, K. M., Gay, G., & Horkoff, J. (2023) · Requirements Engineering · doi:10.1007/s00766-022-00395-3 · Cited by 75
What’s up with Requirements Engineering for Artificial Intelligence Systems?
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., & Grundy, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00008 · Cited by 61
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 70
Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
Windmann, A., Wittenberg, P., Schieseck, M., & Niggemann, O. (2024) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin58382.2024.10774364 · Cited by 58
Software Architecture Challenges for ML Systems
Lewis, G. A., Özkaya, İ., & Xu, X. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00071 · Cited by 57
Subramonyam, H., Im, J., Seifert, C. M., & Adar, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517537 · Cited by 69
Towards Guidelines for Assessing Qualities of Machine Learning Systems
Siebert, J., Jöckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-58793-2_2 · Cited by 60
DevOps for AI – Challenges in Development of AI-enabled Applications
Lwakatare, L. E., Crnković, I., & Bosch, J. (2020) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom50211.2020.9238323 · Cited by 55
Self-citationCharacterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study
Bogner, J., Verdecchia, R., & Gerostathopoulos, I. (2021) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1109/techdebt52882.2021.00016 · Cited by 64
Real-time End-to-End Federated Learning: An Automotive Case Study
Zhang, H., Bosch, J., & Olsson, H. H. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00070 · Cited by 54
Self-citationTowards an AI‐driven business development framework: A multi‐case study
John, M. M., Olsson, H. H., & Bosch, J. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2432 · Cited by 58
Self-citationHabibullah, K. M., & Horkoff, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00009 · Cited by 5
Habiba, U.-E., Haug, M., Bogner, J., & Wagner, S. (2024) · Requirements Engineering · doi:10.1007/s00766-024-00432-3 · Cited by 51
Recupito, G., Pecorelli, F., Catolino, G., Lenarduzzi, V., Taibi, D., Nucci, D. D., & Palomba, F. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112151 · Cited by 58
Going digital: Disruption and transformation in software‐intensive embedded systems ecosystems
Olsson, H. H., & Bosch, J. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2249 · Cited by 45
Self-citationAdapting Software Architectures to Machine Learning Challenges
Serban, A., & Visser, J. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00029 · Cited by 49
Bosch, J., & Olsson, H. H. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2333 · Cited by 42
Self-citationFederated Learning Systems: Architecture Alternatives
Zhang, H., Bosch, J., & Olsson, H. H. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00047 · Cited by 41
Self-citationCharacterizing and Detecting Mismatch in Machine-Learning-Enabled Systems
Lewis, G. A., Bellomo, S., & Özkaya, İ. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00028 · Cited by 41
Gunda, S. K., Yettapu, S. D. R., Bodakunti, S., & Bikki, S. B. (2023) · International Journal of Artificial Intelligence Data Science and Machine Learning · doi:10.63282/3050-9262.ijaidsml-v4i1p112 · Cited by 40
Software Quality for AI: Where We Are Now?
Lenarduzzi, V., Lomio, F., Moreschini, S., Taibi, D., & Tamburri, D. A. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-65854-0_4 · Cited by 41
Model driven engineering for machine learning components: A systematic literature review
Naveed, H., Arora, C., Khalajzadeh, H., Grundy, J., & Haggag, O. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107423 · Cited by 38
von Wangenheim, C. G., Alves, N. D. C., Rauber, M. F., Hauck, J. C. R., & Yeter, I. H. (2021) · Informatics in Education · doi:10.15388/infedu.2022.18 · Cited by 39
Recent Advances in Data Engineering for Networking
Zeydan, E., & Mangues‐Bafalluy, J. (2022) · IEEE Access · doi:10.1109/access.2022.3162863 · Cited by 41
Architectural Design Decisions for Machine Learning Deployment
Warnett, S. J., & Zdun, U. (2022) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa53651.2022.00017 · Cited by 38
Multilayered review of safety approaches for machine learning-based systems in the days of AI
Dey, S., & Lee, S.-W. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110941 · Cited by 42
Rabe, M., Milz, S., & Mäder, P. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00023 · Cited by 39
Maintainability and Scalability in Machine Learning: Challenges and Solutions
Shivashankar, K., Hajj, G. S. A., & Martini, A. (2025) · ACM Computing Surveys · doi:10.1145/3736751 · Cited by 35
John, M. M., Olsson, H. H., & Bosch, J. (2020) · International Conference on Software and System Process · doi:10.1145/3379177.3388892 · Cited by 38
Self-citationOperationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34
Benchmarking Machine Learning Solutions in Production
Cardoso Silva, L., Rezende Zagatti, F., Silva Sette, B., Nildaimon dos Santos Silva, L., Lucredio, D., Furtado Silva, D., & de Medeiros Caseli, H. (2020) · 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla51294.2020.00104 · Cited by 33
Trustworthy artificial intelligence: A decision‐making taxonomy of potential challenges
Akbar, M., Khan, A., Mahmood, S., Rafi, S., & Demi, S. (2023) · Software, Practice & Experience · doi:10.1002/spe.3216 · Cited by 29
Identifying concerns when specifying machine learning-enabled systems: A perspective-based approach
Villamizar, H., Kalinowski, M., Lopes, H., & Méndez, D. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112053 · Cited by 33
Architectural Design Decisions for the Machine Learning Workflow
Warnett, S. J., & Zdun, U. (2022) · Computer · doi:10.1109/mc.2021.3134800 · Cited by 32
Dhanekula, A., & Miah, M. R. (2022) · American Journal of Interdisciplinary Studies · doi:10.63125/707veh84 · Cited by 30
Software engineering practices for machine learning — Adoption, effects, and team assessment
Serban, A., van der Blom, K., Hoos, H. H., & Visser, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111907 · Cited by 34
Requirements Engineering in Machine Learning Projects
Gjorgjevikj, A., Mishev, K., Antovski, L., & Trajanov, D. (2023) · IEEE Access · doi:10.1109/access.2023.3294840 · Cited by 33
A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks
Casey, B., Santos, J. C. S., & Perry, G. (2025) · ACM Computing Surveys · doi:10.1145/3721977 · Cited by 28
Emerging and Changing Tasks in the Development Process for Machine Learning Systems
Liu, H., Eksmo, S., Risberg, J., & Hebig, R. (2020) · International Conference on Software and System Process · doi:10.1145/3379177.3388905 · Cited by 24
Challenges in developing and deploying AI in the engineering, procurement and construction industry
Джусупова, Р., Bosch, J., & Olsson, H. H. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00167 · Cited by 22
Self-citationChapter 1 Climbing the Stairway to Heaven
Olsson, H. H., Alahyari, H., & Bosch, J. (2012) · Springer eBooks · doi:10.1007/978-3-031-10873-0_2 · Cited by 22
Self-citationRequirements and software engineering for automotive perception systems: an interview study
Habibullah, K. M., Heyn, H.-M., Gay, G., Horkoff, J., Knauss, E., Borg, M., Knauss, A., Sivencrona, H., & Li, P. J. (2024) · Requirements Engineering · doi:10.1007/s00766-023-00410-1 · Cited by 21
A Multiple Case Study of Artificial Intelligent System Development in Industry
Nguyen‐Duc, A., Sundbø, I., Nascimento, E., Conte, T. U., Ahmed, I., & Abrahamsson, P. (2020) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3383219.3383220 · Cited by 19
A Holistic Framework for AI Systems in Industrial Applications
Kaymakci, C., Wenninger, S., & Sauer, A. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86797-3_6 · Cited by 19
Practices for Managing Machine Learning Products: A Multivocal Literature Review
Alves, I., Leite, L., Meirelles, P., Kon, F., & Rocha, C. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3287759 · Cited by 18
R, D., & Mathew, J. (2025) · Advances in Science and Technology – Research Journal · doi:10.12913/22998624/197406 · Cited by 18
Effects of Artificial Intelligence on Enterprise Architectures - A Structured Literature Review
Rittelmeyer, J. D., & Sandkuhl, K. (2021) · IEEE International Enterprise Distributed Object Computing Conference Workshops (EDOCW) · doi:10.1109/edocw52865.2021.00042 · Cited by 18
An empirical study on ML DevOps adoption trends, efforts, and benefits analysis
Rzig, D. E., Hassan, F., & Kessentini, M. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107037 · Cited by 18
AI Engineering: Realizing the Potential of AI
Bosch, J., Olsson, H. H., Brinne, B., & Crnković, I. (2022) · IEEE Software · doi:10.1109/ms.2022.3199621 · Cited by 19
Self-citationLandscape of Requirements Engineering for Machine Learning-based AI Systems
Yoshioka, N., Husen, J. H., Tun, H. T., Chen, Z., Washizaki, H., & Fukazawa, Y. (2021) · doi:10.1109/apsecw53869.2021.00011 · Cited by 16
Common challenges of deep reinforcement learning applications development: an empirical study
Morovati, M. M., Tambon, F., Taraghi, M., Nikanjam, A., & Khomh, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10500-5 · Cited by 16
Enabling efficient and low-effort decentralized federated learning with the EdgeFL framework
Zhang, H., Bosch, J., & Olsson, H. H. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107600 · Cited by 19
Self-citationState-of-the-Art and Challenges of Engineering ML- Enabled Software Systems in the Deep Learning Era
Assres, G. M., Bhandari, G. P., Shalaginov, A., Grønli, T., & Ghinea, G. (2025) · ACM Computing Surveys · doi:10.1145/3731597 · Cited by 16
What Machine Learning Can Learn From Software Modularity
Kriens, P., & Verbelen, T. (2022) · Computer · doi:10.48550/arxiv.1906.10366 · Cited by 15
Preliminary Systematic Literature Review of Machine Learning System Development Process
Watanabe, Y., Washizaki, H., Sakamoto, K., Saito, D., Honda, K., Tsuda, N., Fukazawa, Y., & Yoshioka, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05528 · Cited by 15
Making Sense of AI Systems Development
Dolata, M., & Crowston, K. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3338857 · Cited by 18
Time for AI (Ethics) Maturity Model Is NowPreprint
Vakkuri, V., Jantunen, M., Halme, E., Kemell, K.-K., Nguyen-Duc, A., Mikkonen, T., & Abrahamsson, P. (2021) · SafeAI@AAAI · doi:10.48550/arxiv.2101.12701 · Cited by 17
Research challenges for machine learning-constructed software
Pahl, C. (2022) · Service Oriented Computing and Applications · doi:10.1007/s11761-022-00352-6 · Cited by 15
Ibarra-Torres, F., Caiza, G. J., García, M. V., & Barona-Pico, V. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.05.122 · Cited by 14
Towards Human-Centric Psychomotor Recommender Systems
Portaz, M., Manjarrés, Á., & Santos, O. (2023) · User Modeling, Adaptation, and Personalization · doi:10.1145/3563359.3596993 · Cited by 10
Papagiannidis, E., Mikalef, P., Krogstie, J., & Conboy, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15342-6_5 · Cited by 13
Towards a Taxonomy of Cognitive RPA Components
Martínez-Rojas, A., Barba, I., & Enríquez, J. G. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-58779-6_11 · Cited by 12
Data sovereignty for AI pipelines
Altendeitering, M., Pampus, J., Larrinaga, F., Legaristi, J., & Howar, F. M. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528593 · Cited by 12
Human-Centered Recommender Systems
Portaz, M., Riesco, Á. M., & Santos, O. C. (2025) · Human-computer interaction series · doi:10.1007/978-3-031-61375-3_11 · Cited by 11
Parida, S. K., Gerostathopoulos, I., & Bogner, J. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00014 · Cited by 11
The Product Beyond the Model - An Empirical Study of Repositories of Open-Source ML Products
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00006 · Cited by 17
Rauber, M. F., von Wangenheim, C. G., Barbetta, P. A., Borgatto, A. F., Martins, R. M., & Hauck, J. C. R. (2023) · Informatics in Education · doi:10.15388/infedu.2024.10 · Cited by 11
Hidden Feedback Loops in Machine Learning Systems: A Simulation Model and Preliminary Results
Khritankov, A. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-65854-0_5 · Cited by 13
AI Deployment Architecture: Multi-Case Study for Key Factor Identification
John, M. M., Olsson, H. H., & Bosch, J. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00048 · Cited by 14
Self-citationJentzsch, S. F., & Hochgeschwender, N. (2019) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew.2019.00025 · Cited by 11
Machine Learning for Software Engineering: A Systematic MappingPreprint
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13299 · Cited by 11
Anomaly analytics in data-driven machine learning applications
Azimi, S., & Pahl, C. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00593-y · Cited by 10
EdgeFL: A Lightweight Decentralized Federated Learning Framework
Zhang, H., Bosch, J., & Olsson, H. H. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00081 · Cited by 10
Self-citationInterplay of Machine Learning and Software Engineering for Quality Estimations
Abubakar, H., Obaidat, M. S., Gupta, A., Bhattacharya, P., & Tanwar, S. (2020) · International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI) · doi:10.1109/ccci49893.2020.9256507 · Cited by 10
AF-DNDF: Asynchronous Federated Learning of Deep Neural Decision Forests
Zhang, H., Bosch, J., Olsson, H. H., & Koppisetty, A. C. (2021) · 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00047 · Cited by 10
Self-citationNaming the Pain in machine learning-enabled systems engineering
Kalinowski, M., Méndez, D., Giray, G., Alves, A. P. S., Azevedo, K., Escovedo, T., Villamizar, H., Lopes, H., Baldassarre, M. T., Wagner, S., et al. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107866 · Cited by 10
Continuous experimentation on artificial intelligence software: a research agenda
Nguyen‐Duc, A., & Abrahamsson, P. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417039 · Cited by 10
Metadata
- Authors
- Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., & Crnkovic, I.
- Affiliations
- Chalmers University of Technology (SE)
- Malmö University (SE)
- Corresponding author
- Lwakatare, Lucy Ellen
- Venue
- Lecture Notes in Business Information Processing
- Publisher
- Springer International Publishing
- Published
- 2019
- Pages
- 227-243
- ISSN
- 1865-1348 (ISSN-L)
- 1865-1356
- OpenAlex
- W2944264045
- Cited by
- 214
- References
- 22
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 27 · 2023: 31 · 2022: 43 · 2021: 44 · 2020: 29 · 2019: 10
- FWCI
- 25.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Lecture notes in business information processing (publishedVersion, cc-by)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Malmö University Publications (Malmö University) (submittedVersion)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- software engineering challenges, machine learning systems, artificial intelligence systems, software-intensive systems, industrial software development, applied machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
