The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation
Open accessAbstract
Artificial intelligence enabled systems have been an inevitable part of everyday life. However, efficient software engineering principles and processes need to be considered and extended when developing AI- enabled systems. The objective of this study is to identify and classify software engineering challenges that are faced by different companies when developing software-intensive systems that incorporate machine learning components. Using case study approach, we explored the development of machine learning systems from six different companies across various domains and identified main software engineering challenges. The challenges are mapped into a proposed taxonomy that depicts the evolution of use of ML components in software-intensive system in industrial settings. Our study provides insights to software engineering community and research to guide discussions and future research into applied machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
In the citing paper
“Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models [6,7,12,28,34].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862
(2015) · neural information processing systems · Cited by 912
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590
Guest Editors' Introduction: On Applied Research in Machine Learning
Provost, F. J., & Kohavi, R. (1998) · Machine Learning · doi:10.1023/a:1007442505281 · Cited by 385
Software Engineering Challenges of Deep Learning
Arpteg, A., Brinne, B., Crnkovic-Friis, L., & Bosch, J. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00018 · Cited by 238
Self-citationLarge-scale machine learning at twitter
Lin, J., & Kolcz, A. (2012) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2213836.2213958 · Cited by 231
Data Management Challenges in Production Machine Learning
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3054782 · Cited by 214
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184
Trials and tribulations of developers of intelligent systems: A field study
Hill, C., Bellamy, R., Erickson, T. D., & Burnett, M. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739680 · Cited by 115
Cascade Ranking for Operational E-commerce Search
Liu, S., Xiao, F., Ou, W., & Si, L. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098011 · Cited by 111
Research works printed but not linked: 1
- Lefortier, D., Truchet, A., de Rijke, M.: Sources of variability in large-scale machine learning systems. In: Machine Learning Systems (NIPS 2015 Workshop) (2015)
Cited by
Sorted by citations
Items 201–220 of 220
Chapter 6 Lightweight Consistency Checking for Agile Model-Based Development in Practice
Jongeling, R., Ciccozzi, F., Cicchetti, A., & Carlson, J. (2019) · Springer eBooks · doi:10.1007/978-3-031-10873-0_8
Josphineleela, R. (2024) · Communications on Applied Nonlinear Analysis · doi:10.52783/cana.v31.523
Development of an Effective Safety Assessment Capable of Measuring the Safety of AI-Based Systems.
Nyakundi, N. B. N., & Reza, H. (2024) · IEEE International Conference on Electro Information Technology · doi:10.1109/eit60633.2024.10609839
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2024) · Electronics · doi:10.3390/electronics13183716 · Cited by 1
Meisenbacher, S., Galenzowski, J., Förderer, K., Süß, W., Waczowicz, S., Mikut, R., & Hagenmeyer, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-74738-0_18
Designing Secure AI-based Systems: a Multi-Vocal Literature Review
Schneider, S., Saha, A., Mezzi, E., Tuma, K., & Scandariato, R. (2024) · IEEE Secure Development Conference (SecDev) · doi:10.1109/secdev61143.2024.00007
Why Digitalization Will Kill Your Company Too
Bosch, J., & Olsson, H. H. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-89277-6_1
Self-citationAn Approach for Integrated Development of an MLOps Architecture
Heck, P., Snoeren, J., Veracx, M., & Peeters, M. W. H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02138-0_1
Diffusion Model of Anti-Patterns for Machine Learning Projects
Takeuchi, H., Kaiya, H., Nakagawa, H., & Ogata, S. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.153
Virvou, M., Tsihrintzis, G. A., & Tanabe, Y. (2025) · Learning and analytics in intelligent systems · doi:10.1007/978-3-031-98410-5_1
Chidum, P., & AMADI, A. O. (2026) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.07.0226.0352
Architectural Design Decisions for Monitoring Deployed Reinforcement Learning Systems
Fang, Z., Warnett, S. J., & Zdun, U. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39236-7_20
From ITIL to AIOps in Public Sector: A Systematic Literature Review
Ganduri, C. (2026) · Journal of Engineering Research and Sciences · doi:10.55708/js0506005
Machine Learning Lifecycle: A Survey
Kosmas, I., Papadopoulos, T., & Michalakelis, C. (2026) · AppliedMath · doi:10.3390/appliedmath6070113
A Governance Framework For Trustworthy AI-Generated Data Engineering Artifacts In Cloud Environments
Kodati, P. K. (2026) · International Journal of Artificial Intelligence and Machine Learning · doi:10.51483/ijaiml.6.8s.2026.86-96
От прототипа к продакшену: инженерные вызовы внедрения AI-систем
Токсынов, Ш. М. (2025) · True Masters Innovations · doi:10.65769/ls6ih5buxg
Towards Real-time Object Detection for Safety Analysis in an ML-Enabled System Simulation
Adigun, J. G., Aschenbrenner, P., & Felderer, M. (2024) · WiPiEC Journal - Works in Progress in Embedded Computing Journal · doi:10.64552/wipiec.v10i2.69
Metadata
- Authors
- Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., & Crnkovic, I.
- Affiliations
- Chalmers University of Technology (SE)
- Malmö University (SE)
- Corresponding author
- Lwakatare, Lucy Ellen
- Venue
- Lecture Notes in Business Information Processing
- Publisher
- Springer International Publishing
- Published
- 2019
- Pages
- 227-243
- ISSN
- 1865-1348 (ISSN-L)
- 1865-1356
- OpenAlex
- W2944264045
- Cited by
- 214
- References
- 22
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 27 · 2023: 31 · 2022: 43 · 2021: 44 · 2020: 29 · 2019: 10
- FWCI
- 25.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Lecture notes in business information processing (publishedVersion, cc-by)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Malmö University Publications (Malmö University) (submittedVersion)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- software engineering challenges, machine learning systems, artificial intelligence systems, software-intensive systems, industrial software development, applied machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
