CoChat
Sign Up
Cited by 187Software Engineering Techniques and PracticesSoftware Engineering ResearchAdvanced Software Engineering Methodologies

The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index; none of the 326 sources its claims cite is checked for support yet.

A software engineering perspective on engineering machine learning systems: State of the art and challenges

Open access
Giray, Görkem·2021·Journal of Systems and Software
DOI 10.1016/j.jss.2021.111031
Open this paper in CoChat — free accountOpen PDF

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic literature review synthesizes 141 SE-venue primary studies (2007–Dec 2020) on engineering machine learning systems. It finds that ML systems’ non-deterministic nature complicates all SE aspects, no SE aspect yet has mature tools and techniques, and testing is the most studied area but still lacks strong experimental proof.

Key findings

  • The review analyzed 141 primary studies from SE venues; research interest increased sharply from 2018 onward, and 91% of studies appeared in conferences/workshops (129 papers) versus 12 in journals.
  • Testing is by far the most popular area among researchers, but even for testing ML systems engineers have only tool prototypes and solution proposals with weak experimental proof; many challenges were identified through surveys and interviews.
  • Among 93 experiment/case-study papers, 65 (70%) used only DL algorithms, 19 used traditional ML, 4 used both, and 5 did not report algorithms; image classification was the most popular application scenario, used in 57 studies.
  • No SE aspect has a mature set of tools and techniques, and the paper calls for experiments and case studies, ideally in industrial environments, to better understand challenges and propose solutions.

Methods

  • Systematic literature review following established guidelines, with database search (ACM, IEEE Xplore, ScienceDirect, Springer, Wiley, Google Scholar), backward/forward snowballing, and manual search of ICSE, ESEC/FSE, AIRE, and RAISE.
  • Inclusion/exclusion criteria restricted papers to SE venues and to those involving at least one SE challenge of engineering ML systems; after de-duplication and quality assessment, the final pool was 141 studies.
  • Quantitative analysis reported frequencies/percentages for research methods and application scenarios/datasets; qualitative open coding analyzed challenges and solutions, classified under SWEBOK-based SE knowledge areas.

Limitations

  • Paper-stated: scope is limited to studies published until 31 December 2020, peer-reviewed academic venues (grey literature excluded), and SE venues only (AI/ML/DL/Data Science/Data Management venues excluded); primary study selection, data extraction, and synthesis are subject to researcher bias, which may be higher in a single-author SLR.
  • Paper-stated: single data extraction can produce more errors than double data extraction, and the author notes that selection only from SE venues may miss relevant studies from other venues.
  • Evident from text: the search used broad terms and a “search saturation effect,” checking only the first 200 results per query in each database unless results 190–200 still looked relevant, which may limit recall despite snowballing and manual search.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.jss.2021.111031

Cites

256 of 256 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 326 cited sources checked · 0 supported · 0 flagged

Show all 143 claims

  • It may state two separate claims as one. 62
  • Probably not a substantial claim. 155
  • The quote only partly states this claim. 1
192 of 192

  • The authors state this as an assumption.

    The search space can be restricted by assuming a search saturation effect, such that checking the first 200 results of each query in each database is sufficient.

    “Therefore, as others have done (Adams et al., 2016;Garousi and Mäntylä, 2016;Garousi et al., 2018;Godin et al., 2015), I had to restrict the search space by assuming a”

  • The paper relies on this without support.

    The primary studies' abstracts are representative of the full papers for the purpose of term frequency analysis.

    “I created a bar chart of the most frequently used 50 terms in the primary studies' abstracts (Figure 5).”

  • The paper relies on this without support.

    The classification of SE challenges based on the main SE knowledge areas presented in Wan et al. (2019) is suitable for organizing the identified challenges.

    “The identified SE challenges are classified based on the main SE knowledge areas presented in (Wan et al., 2019).”

  • The paper relies on this without support.

    The outputs of an ML model may be used by many other components implicitly.

    “The outputs of an ML model may be used by many other components implicitly.”

  • The paper relies on this without support.

    ML system performance is subject to change due to its non-deterministic nature.

    “Moreover, ML systems should be monitored continuously since their performance is subject to change due to their non-deterministic nature.”

  • The paper relies on this without support.

    ML models are dependent on ML frameworks.

    “ML models are also dependent on ML frameworks.”

  • The paper relies on this without support.

    Some large-scale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.

    “While it is necessary to keep track of changes in data and ML models, some largescale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.”

  • The paper relies on this without support.

    Each experiment may have many components that affect the outcome, i.e., the ML model.

    “Each experiment may have many components that affect the outcome, i.e., the ML model.”

  • The paper relies on this without support.

    It is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.

    “Also, it is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.”

  • The paper relies on this without support.

    One of the hardest parts of maintaining ML systems is to keep the performance (such as accuracy) at a certain level or improve it if needed.

    “One of the hardest parts of maintaining ML systems is to keep the performance (such as accuracy) at a certain level or improve it if needed.”

  • The paper relies on this without support.

    There are some process models, phases, activities that document steps to develop and maintain ML models.

    “There are some process models, phases, activities that document steps to develop and maintain ML models.”

  • The paper relies on this without support.

    Adding ML components to a software system involves additional actions to be performed.

    “Adding ML components to a software system involves additional actions to be performed.”

  • The paper relies on this without support.

    Having the required people with proper skill sets is necessary but not sufficient to engineer ML systems.

    “Having the required people with proper skill sets is necessary but not sufficient to engineer ML systems.”

  • The paper relies on this without support.

    Specifying ML components' requirements requires a different perspective than traditional software components.

    “Specifying ML components' requirements requires a different perspective than traditional software components.”

  • The paper relies on this without support.

    ML systems include traditional software components as well.

    “Besides, ML systems include traditional software components as well.”

  • The paper relies on this without support.

    Developers deal with additional complications while developing ML systems.

    “Developers deal with additional complications while developing ML systems.”

  • The paper relies on this without support.

    Another challenge is the estimation of resource requirements for production environment.

    “Another challenge is the estimation of resource requirements for production environment.”

  • The paper relies on this without support.

    Data, ML models, and experiments have become essential configuration items in maintaining ML systems.

    “Data, ML models, and experiments have become essential configuration items in maintaining ML systems.”

  • The paper relies on this without support.

    Keeping versions of data and ML models require tools integrated with the development environment.

    “Keeping versions of data and ML models require tools integrated with the development environment.”

  • The paper relies on this without support.

    Engineering ML systems is a highly practical area.

    “although engineering ML systems is a highly practical area”

  • The paper relies on this without support.

    ML systems engineering brings together two challenging research areas, i.e., SE and ML, and requires various skillsets.

    “ML systems engineering brings together two challenging research areas, i.e., SE and ML, and requires various skillsets.”

  • The paper relies on this without support.

    SE venues can provide a representative set to identify state of the art and challenges of engineering ML systems.

    “However, I believe that SE venues can provide a representative set to identify state of the art and challenges of engineering ML systems.”

  • The paper relies on this without support.

    More and more software systems are involving ML capabilities.

    “More and more software systems are involving ML capabilities.”

  • The paper relies on this without support.

    Advances in machine learning lead to a transition from conventional software development, where algorithms are hard-coded by humans, to ML systems that are materialized through learning from data.

    “Advances in machine learning lead to a transition from the conventional view of software development, where algorithms are hard-coded by humans, to ML systems that are materialized through learning from data.”
24 of 24

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Online testing is said both to identify and miss the same violations

    Minor

    The sentence says that safety violations identified through online testing could not be identified through online testing. The surrounding comparison concerns offline versus online testing.

  • Review method is labeled both SMS and SLR

    Minor

    Table 1 classifies this study as a systematic mapping study (SMS), while the abstract says it was a systematic literature review (SLR).

  • Number of studies using multiple methods is misstated

    Minor

    The text names at least four studies using multiple research methods before claiming that only three such studies account for the total in Figure 6.

  • Test management is described as unaddressed despite a reported case study

    Minor

    The research-directions section says researchers have not addressed test management, but the results section reports a case study that used tools and spreadsheets to manage testing.

4 of 4

Citations

Cites

Items 201–256 of 256

Flagged first, then by citations

Research works printed but not linked: 6

  1. Fujii, 2020, Guidelines for quality assurance of machine learning-based artificial intelligence, Int. J. Softw. Eng. Knowl. Eng., 1
  2. Wan, 2019, How does machine learning change software development practices?, IEEE Trans. Softw. Eng.
  3. Fei-Fei, L., 2010. ImageNet: crowdsourcing, benchmarking & other cool things. In: CMU VASC Seminar (Vol. 16). March. pp. 18-25.
  4. Schwerdtner, P., Grezner, F., Kapoor, N., Assion, F., Sass, R., Günther, W., Hüger, F., Schlicht, P., 2020. Risk assessment for machine learning models. In: Machine Learning for Autonomous Driving Workshop at the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
  5. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R., 2014. Intriguing properties of neural networks. In: 2nd International Conference on Learning Representations (ICLR).
  6. Zaharia, 2018, Accelerating the machine learning lifecycle with mlflow, IEEE Data Eng. Bull., 41, 39

Cited by

Sorted by citations

See all 187

Related

See all

Metadata

Authors
Giray, G.
Corresponding author
Giray, Görkem
ORCID
Giray, Görkem
Venue
Journal of Systems and Software
Publisher
Elsevier BV
Published
2021-10
Volume
180
Pages
111031
ISSN
0164-1212 (ISSN-L)
DOI
10.1016/j.jss.2021.111031
OpenAlex
W3112475255
Cited by
187
References
256
Citations by year
2026: 23 · 2025: 37 · 2024: 38 · 2023: 43 · 2022: 36 · 2021: 10
FWCI
28.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Systems and Software (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
software engineering for machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review