The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation
Open accessAbstract
Artificial intelligence enabled systems have been an inevitable part of everyday life. However, efficient software engineering principles and processes need to be considered and extended when developing AI- enabled systems. The objective of this study is to identify and classify software engineering challenges that are faced by different companies when developing software-intensive systems that incorporate machine learning components. Using case study approach, we explored the development of machine learning systems from six different companies across various domains and identified main software engineering challenges. The challenges are mapped into a proposed taxonomy that depicts the evolution of use of ML components in software-intensive system in industrial settings. Our study provides insights to software engineering community and research to guide discussions and future research into applied machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
In the citing paper
“Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models [6,7,12,28,34].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862
(2015) · neural information processing systems · Cited by 912
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590
Guest Editors' Introduction: On Applied Research in Machine Learning
Provost, F. J., & Kohavi, R. (1998) · Machine Learning · doi:10.1023/a:1007442505281 · Cited by 385
Software Engineering Challenges of Deep Learning
Arpteg, A., Brinne, B., Crnkovic-Friis, L., & Bosch, J. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00018 · Cited by 238
Self-citationLarge-scale machine learning at twitter
Lin, J., & Kolcz, A. (2012) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2213836.2213958 · Cited by 231
Data Management Challenges in Production Machine Learning
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3054782 · Cited by 214
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184
Trials and tribulations of developers of intelligent systems: A field study
Hill, C., Bellamy, R., Erickson, T. D., & Burnett, M. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739680 · Cited by 115
Cascade Ranking for Operational E-commerce Search
Liu, S., Xiao, F., Ou, W., & Si, L. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098011 · Cited by 111
Automatically Tracking Metadata and Provenance of Machine Learning Experiments
Schelter, S., Böse, J.-H., Kirschnick, J., Klein, T., & Seufert, S. (2017) · Cited by 78
Attenberg, J., & Provost, F. J. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964906 · Cited by 77
An Approach to Software Testing of Machine Learning Applications
Murphy, C., Kaiser, G. E., & Arias, M. (2007) · doi:10.7916/d8r49znr · Cited by 61
Tata, S., Popescul, A., Najork, M., Colagrosso, M., Gibbons, J., Green, A. I., Mah, A., Smith, M. D., Garg, D., Meyer, C., et al. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098048 · Cited by 52
Practical Machine Learning for Cloud Intrusion Detection
Kumar, R. S. S., Wicker, A. W., & Swann, M. (2017) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3128572.3140445 · Cited by 44
It takes three to tango: Requirement, outcome/data, and AI driven development
Bosch, J., Olsson, H. H., & Crnković, I. (2018) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 38
Self-citationA Framework for Adoption of Machine Learning in Industry for Software Defect Prediction
Rana, R., Staron, M., Hansson, J., Nilsson, M., & Meding, W. (2014) · International Conference on Software Engineering and Applications (ICSOFT-EA) · doi:10.5220/0005099303830392 · Cited by 37
Design principles of massive, robust prediction systems
Raeder, T., Stitelman, O., Dalessandro, B., Perlich, C., & Provost, F. J. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339740 · Cited by 32
Versioning for End-to-End Machine Learning Pipelines
van der Weide, T., Papadopoulos, D., Smirnov, O., Zielinski, M., & van Kasteren, T. L. M. (2017) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3076246.3076248 · Cited by 28
Hains, G., Jakobsson, A., & Khmelevsky, Y. (2018) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon.2018.8369576 · Cited by 28
An Architecture for Agile Machine Learning in Real-Time Applications
Schleier-Smith, J. (2015) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2788628 · Cited by 21
Sjafrie, H. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429316777-2 · Cited by 1
Research works printed but not linked: 1
- Lefortier, D., Truchet, A., de Rijke, M.: Sources of variability in large-scale machine learning systems. In: Machine Learning Systems (NIPS 2015 Workshop) (2015)
Cited by
Sorted by citations
Perifanis, N.-A., & Kitsios, F. (2023) · Information · doi:10.3390/info14020085 · Cited by 407
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 426
Taxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380395 · Cited by 299
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Self-citationGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metadata
- Authors
- Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., & Crnkovic, I.
- Affiliations
- Chalmers University of Technology (SE)
- Malmö University (SE)
- Corresponding author
- Lwakatare, Lucy Ellen
- Venue
- Lecture Notes in Business Information Processing
- Publisher
- Springer International Publishing
- Published
- 2019
- Pages
- 227-243
- ISSN
- 1865-1348 (ISSN-L)
- 1865-1356
- OpenAlex
- W2944264045
- Cited by
- 214
- References
- 22
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 27 · 2023: 31 · 2022: 43 · 2021: 44 · 2020: 29 · 2019: 10
- FWCI
- 25.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Lecture notes in business information processing (publishedVersion, cc-by)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Malmö University Publications (Malmö University) (submittedVersion)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- software engineering challenges, machine learning systems, artificial intelligence systems, software-intensive systems, industrial software development, applied machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
