CoChat
Sign Up
Cited by 81Software Engineering Techniques and PracticesSoftware Engineering ResearchAdvanced Software Engineering Methodologies

Software engineering challenges for machine learning applications: A literature review

Open access
Kumeno, Fumihiro·2020·Intelligent Decision Technologies
DOI 10.3233/idt-190160
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning techniques, especially deep learning, have achieved remarkable breakthroughs over the past decade. At present, machine learning applications are deployed in many fields. However, the outcomes of software engineering researches are not always easily utilized in the development and deployment of machine learning applications. The main reason for this difficulty is the many differences between machine learning applications and traditional information systems. Machine learning techniques are evolving rapidly, but face inherent technical and non-technical challenges that complicate their lifecycle activities. This review paper attempts to clarify the software engineering challenges for machine learning applications that either exist or potentially exist by conducting a systematic literature collection and by mapping the identified challenge topics to knowledge areas defined by the Software Engineering Body of Knowledge (Swebok).

AI snapshot

Written from the paper’s full text.

TL;DR

This literature review systematically collects papers on software engineering challenges for machine learning applications and maps the identified challenge topics to SWEBOK 3.0 knowledge areas. It finds that challenges span many knowledge areas, with safety, security, and V&V recurring across several, and Software Design and Software Construction most represented.

Key findings

  • The review selected 115 papers; 108 were mapped to SWEBOK knowledge areas, while the remaining 7 surveyed domain-specific ML applications such as robotics, medical systems, networking, and machine translation.
  • Software Design was related to the most papers in the mapping process, followed by Software Construction; many survey papers addressed ML-specific techniques and challenges in these two knowledge areas.
  • Safety, security, and V&V for ML applications are major challenge topics across several knowledge areas; the review also identifies ethics and regulations, economic impacts, and risk management as engineering-practice challenges.
  • Publication trends show the first SE-related paper after 2000 was by Senyard et al. in 2003; few papers appeared from 2004 to 2014, but significantly more were published from 2015 onward, while ML-related papers moderately increased from 2008.

Methods

  • The paper collection used a systematic snowballing approach. A start set was generated from frequently appearing SWEBOK knowledge-area keywords combined with ML-related keywords “machine learning,” “deep learning,” and “artificial intelligence” via Google, with inclusion criteria including papers after 2000, journals/conferences/workshops/technical reports including arXiv, and the most recent version.
  • Additional papers were collected using keywords “Model engineering,” “Automated Machine Learning,” “Metamorphic testing,” and “Technical Debt,” followed by two iterations of backward and forward snowballing.
  • After reviewing collected papers, the author identified challenge topics from SE and ML perspectives and formed a relational map between challenges and SWEBOK 3.0 knowledge areas, using chapters 1–12 and excluding foundational chapters 13–15.

Limitations

  • The paper states that it reports preliminary results, that only two iterations of snowballing were performed, and that more iterations would provide more comprehensive results.
  • The paper states that even a comprehensive collection would provide only a snapshot because related papers are published daily.
  • The paper states that additional papers were selected by one researcher and the relation mapping was also conducted by the same researcher; multiple persons must review, select, and map included papers to achieve more objective and persuasive conclusions, and threats to validity must be carefully discussed. Evident from the text: reliance on Google search and keyword extraction, plus single-researcher selection and mapping, may introduce selection and subjectivity bias, and no inter-rater reliability is reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3233/idt-190160

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 150 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    Design challenges for security, safety and V&V of ML applications exist.

    “-Security, Safety and V&V: Design challenges for security, safety and V&V of ML applications [12,13,19,20,27,30,33,35,[47][48][49]61].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12,13,19,20,27,30,33,35,[47][48][49]61]

      Not checked in the free run.

  • Findingnone given

    Software requirements activities for ML applications involve ML-specific activities: data and feasibility analysis, requirements elicitation, requirement specifications, and validation of ML-functions and performances.

    “Software requirements activities for ML applications involve ML-specific activities, namely, data and feasibility analysis, requirements elicitation, requirement specifications, and validation of ML-functions and performances.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    The number of ML-related papers increased by 51 papers after Iteration 2.

    “The number of ML-related papers was significantly increased (by 51 papers) after Iteration 2, because ML techniques anchor ML applications; accordingly, SE-related papers also refer to papers on ML techniques and challenges.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    Fairness is another emerging property of ML applications.

    “Fairness is another emerging property [26,118].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [26] source

      Not checked in the free run.

    • cites [118]

      Not checked in the free run.

  • Findingcitation

    Requirement activities on data-oriented works have brought new challenges.

    “Requirement activities on data-oriented works have brought new challenges.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 34
  • Classified as probably not a substantial claim. 50
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Various topics were related to Software Design and Software Construction, and many survey papers addressed the ML-specific techniques and challenges of these two KAs.

    “Various top- ics were related to Software Design and Software Construction, and many survey papers addressed the MLspecific techniques and challenges of these two KAs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingreasoning

    ML techniques are widely used and are being integrated into mission critical systems; accordingly, safety, security and V&V have become critical issues.

    “ML techniques are widely used and are being integrated into mission critical systems; accordingly, safety, security and V&V (validation and verification) has become critical issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the first phase of the paper collection (Start Set), 13 papers were selected: 12 SE-related papers and one ML-related paper.

    “In the first phase of the paper collection (Start Set), we selected 13 papers: 12 SE-related papers and one MLrelated paper.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Few papers were published from 2004 to 2014, but significantly more were published from 2015 onward.

    “Few papers were published from 2004 to 2014, but significantly more were published from 2015 onward.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The literature search process yielded 115 papers.

    “The literature search process in the previous section yielded 115 papers (see Table 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The needs to preserve the privacy and safety of sensitive datasets and to ensure legal compliance with a new regulation such as the European General Data Protection Regulation may impact research directions in requirements engineering.

    “Furthermore, the needs to preserve the privacy and safety of sensitive datasets and to ensure legal compliance with a new regulation such as the European General Data Protection Regulation may impact research directions in requirements engineering [18,24,60,105].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

    • cites [60]

      Not checked in the free run.

    • cites [105]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Interpretability as a property of software requirements has emerged with the progress of ML techniques and applications.

    “Interpretability as a property of software requirements has emerged with the progress of ML techniques and applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Challenges on the software structure of ML applications include complex software modules of ML algorithms, anti-patterns in ML applications, and various design issues in ML models (model selection, customization and reuse).

    “-Software Structure: Challenges on the software structure of ML applications.This category includes the complex software modules of ML algorithms [14], anti-patterns in ML applications [17], and various design issues in ML models (model selection, customization and reuse) [16,34,58,59].”
    Quote in the paperSubstantial claim0 of 6 sources checked
    • cites [14] source

      Not checked in the free run.

    • cites [17] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

    • cites [34]

      Not checked in the free run.

    • cites [58]

      Not checked in the free run.

    • cites [59]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    These software requirements activities are difficult because the requirements may change frequently in large-scale systems such as automated vehicles, and they are also very complex.

    “These activities are difficult because the requirements may change frequently in largescale systems such as automated vehicles, they are also very complex [59,90].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [59]

      Not checked in the free run.

    • cites [90]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In Iterations 1 and 2, 35 SE-related papers and 67 ML-related papers were selected.

    “In the following phases (Iterations 1 and 2), we selected 35 SE-related papers and 67 ML-related papers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Software Design was related to the most number of papers in the mapping process, followed by Software Construction.

    “Software Design was related to the most number of papers in the mapping process, followed by Software Construction.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Cost reduction techniques of ML testing, including the cost reduction of traditional methodologies such as search-based testcase generation, test prioritization, and test-case minimization, are a challenge.

    “-Cost Reduction: Cost reduction techniques of ML testing, including the cost reduction of traditional methodologies such as search-based testcase generation, test prioritization, and test-case minimization.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Testing for data validation and data cleaning, and adversarial testing on the training data and data testing to detect privacy violations for designing secure ML applications, can be challenging issues.

    “-Testing Data: Testing for data validation and data cleaning.For designing secure ML applications, adversarial testing on the training data and data testing to detect privacy violations can be challenging issues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Testing of ML specific properties such as overfitting, interpretability and fairness is a challenge.

    “-Testing Properties: Testing of ML specific properties such as overfitting, interpretability and fairness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    In ML applications, the ML models may be frequently retrained with concept drifts and thus change behavior autonomously in unintended ways.

    “In a ML applications, the ML models may be frequently retrained with concept drifts and thus change behavior autonomously in unintended ways.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    There are challenging issues on testing other ML mechanisms such as unsupervised learning, reinforce learning, transfer learning and meta-learning, while many current researches focus on supervised learning.

    “-Testing ML techniques: Many of current researches focus on supervised learning.There are challenging issues on testing other ML mechanisms such as unsupervised learning, reinforce learning, transfer learning and meta-learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    There are needs of designing tools for ML applications, such as tools for non-expert ML designers, visualization tools for understanding the relationships between data and the behavior of algorithms, tools for ensuring interoperability with other tools, and domain-specific language (DSL) support.

    “-Tools: Needs of designing tools for ML applications, such as tools for non-expert ML designers, visualization tools for understanding the relationships between data and the behavior of algorithms, tools for ensuring interoperability with other tools, and domain-specific language (DSL) support [16,25,42,66].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [16] source

      Not checked in the free run.

    • cites [25] source

      Not checked in the free run.

    • cites [42]

      Not checked in the free run.

    • cites [66]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Selected papers for the Software Construction KA are also related to the Software Design KA, except Islam et al., who reported the challenges facing the use of ML libraries.

    “Selected papers for this KA are also related to the Software Design KA (except Islam et al. [28], who reported the challenges facing the use of ML libraries).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Findingnone given

    Making reliable test oracles with less human intervention for ML applications is a challenge.

    “-Oracle Problem: How to make reliable test oracles with less human intervention for ML applications.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The design and construction of well-performed ML models is time consuming, requires a significant amount of resources and highly specialized experts.

    “-Automated ML (Auto ML): The design and construction of well-performed ML models is time consuming, requires a significant amount of resources and highly specialized experts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    These situations can pose various maintenance challenges of ML applications.

    “These situations can pose various maintenance challenges of ML applications:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The design and construction of reusable testing assets for ML applications is a challenge.

    “-Benchmarks: The design and construction of reusable testing assets for ML applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    In real-world ML applications, uncertain events might occur in the deployment phase.

    “In real-world ML applications, uncertain events might occur in the deployment phase.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Automated machine learning (AutoML) is a new research topic that aims to resolve the problem of time-consuming and resource-intensive ML model design and construction.

    “Automated machine learning (AutoML) is a new research topic that aims to resolve this problem [29,54,55,64,78,109,113].”
    Quote in the paperNot a substantial claim0 of 7 sources checked
    • cites [29] source

      Not checked in the free run.

    • cites [54]

      Not checked in the free run.

    • cites [55]

      Not checked in the free run.

    • cites [64]

      Not checked in the free run.

    • cites [78]

      Not checked in the free run.

    • cites [109]

      Not checked in the free run.

    • cites [113]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    The environment of production ML might largely differ from the environment the ML models were trained and evaluated.

    “The environment of production ML might largely differ from the environment the ML models were trained and evaluated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    These papers did not discuss the challenges of constructing ML applications, but their topics were potentially closely related to construction challenges.

    “These papers did not discuss the challenges of constructing ML applications, but their topics were potentially closely related to construction challenges.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The design and embedding of mutants to improve simulations of real-world ML bugs is a challenge.

    “-Mutation Testing: The design and embedding of mutants to improve simulations of real-world ML bugs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Tool support for the development process can be a further challenge issue.

    “Tool support for the development process can be a further challenge issue.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    The ethics of ML applications relate to issues on safety, privacy and discrimination, and ethical topics should be included in quality evaluations of ML applications.

    “-Ethics and Regulations: The ethics of ML applications relate to issues on safety, privacy and discrimination [20,24,49,59,103,106].Ethical topics should be included in quality evaluations of ML applications.”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [20] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

    • cites [49]

      Not checked in the free run.

    • cites [59]

      Not checked in the free run.

    • cites [103]

      Not checked in the free run.

    • cites [106]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    To operate real-world ML applications, complex data configuration management is indispensable as well as software configuration management.

    “To operate real-world ML applications, the complex data configuration management is indispensable as well as the software configuration management.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The software quality challenges in ML applications embrace various topics.

    “The software quality challenges in ML applications also embrace various topics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    In ML applications, the deployment phase will most likely add new functional modules to the existing system, and the platform and infrastructure might greatly differ from the training and evaluation environment, posing compatibility, portability and scalability challenges.

    “-Operating Environment: In ML applications, the deployment phase will most likely add new functional modules to the existing system.In the deployment and operation phases, the platform and infrastructure of the ML application might greatly differ from the training and evaluation environment of the ML model.These differences pose compatibility, portability and scalability challenges [24,59,107].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [24] source

      Not checked in the free run.

    • cites [59]

      Not checked in the free run.

    • cites [107]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Corbett-Davies et al. proposed a measure for evaluating fairness of ML applications.

    “Corbett-Davies et al. [118] proposed a measure for evaluating fairness of ML applications.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [118]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Estimating the effort of an ML project is challenging because it is difficult to know to what extent the ML model will achieve its goal, and to estimate how many iterations will be needed to reach acceptable performance levels.

    “-Effort Estimation: Estimating the effort of an ML project is challenging because it is difficult to know to what extent the ML model will achieve its goal, and to estimate how many iterations will be needed to reach the state in which the performance gets acceptable levels [18,83].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [83]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Training data, ML models and ML platforms are key components of ML applications, and challenges on the quality of each component have been discussed, including data anomaly, imbalanced or biased data, encrypted data, data evaluation/cleaning, and novelty detection.

    “-Component Quality: Training data, ML models and ML platforms (e.g.scikit-learn [125], Tensor flow [126], Weka [127]) is the key components of ML applications.The challenges on the quality of each components have been discussed.The challenge issues on data quality for ML applications includes data anomaly [23], imbalanced or biased data, encrypted data [60], data evaluation/cleaning [39][40][41]74”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [23] source

      Not checked in the free run.

    • cites [60]

      Not checked in the free run.

    • cites [31] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Mission critical systems in some domains strongly require safety and security, and conforming ML applications to industry standards such as DO-178, ISO26262 and ASIL, and Common Criteria is a difficult but important challenge to realize mission critical ML applications in industry.

    “-Safety and Security: Mission critical systems in some domains are strongly required safety and security.There are industry standards which address safety or security such as DO-178 [128], ISO26262 and ASIL [129] and Common Criteria [130].How to conform ML applications to these standards is difficult but important challenges to realize mission critical ML applications in industry [20,30,35,61,82,”
    Quote in the paperNot a substantial claim0 of 7 sources checked
    • cites [20] source

      Not checked in the free run.

    • cites [30]

      Not checked in the free run.

    • cites [35]

      Not checked in the free run.

    • cites [61]

      Not checked in the free run.

    • cites [82]

      Not checked in the free run.

    • cites [90]

      Not checked in the free run.

    • cites [119]

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Risk management of the development, deployment and operation of ML applications is critical, but is rendered difficult by various uncertainties.

    “-Risk Management: Risk management of the development, deployment and operation of ML applications is critical, but is rendered difficult by various uncertainties [17,24].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [17] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Quality measurement for ML applications is closely related to system safety.

    “Quality measurement for ML applications is closely related to system safety.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    V&V is an integral part of software quality assurance for ML applications, and many papers addressed technical challenges of V&V for ML applications.

    “-Validation & Verification: V&V is the integral part of software quality assurance for ML applications.Many papers addressed technical challenges of V&V for ML applications [13,30,[47][48][49]90,102,119].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13,30,[47][48][49]90,102,119]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Trial-based processes can address the difficulties of making customers understand ML application properties, but further researches are needed to build a solid foundation for the engineering disciplines.

    “Trial-based processes can address these difficulties, but further researches are needed to build a solid foundation for the engineering disciplines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Ishikawa et al. reported the difficulties to make customers better understand the properties of ML applications such as imperfections.

    “Ishikawa et al. [83] reported the difficulties to make customers better understand the properties of ML applications such as imperfections.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [83]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Many researches on ML testing have been published, but many challenges remain and are still emerging.

    “Many researches on ML testing have been published, but many challenges remain and still emerging.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The broad range of challenge topics were extracted through the mapping.

    “The broad range of challenge topics were extracted through the mapping.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The ethics and regulation of ML applications relate to the professionalism of ML applications engineers.

    “The ethics and regulation of ML applications also relate to the professionalism of ML applications engineers (see Section 4.10).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The main challenges in group dynamics and psychology are collaboration to ensure a successful project and adequate communication with customers.

    “The main challenges in this category are collaboration to ensure a successful project and adequate communication with customers [17,18,21,27,40,59,83].”
    Quote in the paperNot a substantial claim0 of 7 sources checked
    • cites [17] source

      Not checked in the free run.

    • cites [18] source

      Not checked in the free run.

    • cites [21] source

      Not checked in the free run.

    • cites [27] source

      Not checked in the free run.

    • cites [40]

      Not checked in the free run.

    • cites [59]

      Not checked in the free run.

    • cites [83]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The main reason for the difficulty of utilizing software engineering research outcomes in ML applications is the many differences between machine learning applications and traditional information systems.

    “The main reason for this difficulty is the many differences between machine learning applications and traditional information systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    The methodologies and tools of software engineering have greatly contributed to traditional information systems, but are difficult to implement in ML application projects because ML applications and traditional software systems differ in fundamental ways.

    “The methodologies and tools of software engineering (SE) have greatly contributed to a wide range of activities in the lifecycles of traditional information systems, but are difficult to implement in ML application projects because ML applications and traditional software systems differ in fundamental ways.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    The outcomes of software engineering researches are not always easily utilized in the development and deployment of machine learning applications.

    “However, the outcomes of software engineering researches are not always easily utilized in the development and deployment of machine learning applications.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The Software Engineering Body of Knowledge (Swebok) classifies software engineering topics into knowledge areas.

    “The Software Engineering Body of Knowledge (Swebok) [6] classifies software engineering topics into knowledge areas.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Swebok's 15 knowledge areas summarize basic concepts and include a reference list pointing to more detailed information.

    “Its 15 knowledge areas (KAs) summarize basic concepts and include a reference list pointing to more detailed information.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Machine learning techniques are evolving rapidly, but face inherent technical and non-technical challenges that complicate their lifecycle activities.

    “Machine learning techniques are evolving rapidly, but face inherent technical and non-technical challenges that complicate their lifecycle activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Machine learning algorithms, models and related techniques are rapidly evolving and new challenges are emerging.

    “Machine learning algorithms, models and related techniques are rapidly evolving and new challenges are emerging.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    SE challenges for ML applications have been discussed in many papers, but no survey paper has clarified the overview of SE challenges for ML applications.

    “SE challenges for ML applications have been discussed in many papers [15][16][17][18]21], but to our knowledge, no survey paper has clarified the overview of SE challenges for ML applications, that is, what SE challenges have been discussed?and which SE research topics are closely related to each challenge?”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15][16][17][18]21]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Swebok Version 3.0 is the most recently published version of the body of knowledge for the field of software engineering.

    “Swebok Version 3.0 [6] is the most recently published version of the body of knowledge for the field of software engineering.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • May combine two claims.

    Premisenone given

    The behavior of an ML model-based program depends on the training data, and is often unpredictable.

    “The behavior of an ML modelbased program depends on the training data, and is often unpredictable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The rapid evolution of ML techniques and emerging challenges make software engineering practices for ML applications more difficult.

    “Such situations make software engineering practices for ML applications more difficult activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These demands have hindered the development of ML applications in industry.

    “These demands have hindered the development of ML applications in industry.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    The Software Construction KA is strongly linked to the Software Design and Software Testing KAs.

    “The Software Construction knowledge area (KA) is linked to all the other KAs, but it is most strongly linked to Software Design and Software Testing because the software construction process involves significant software design and testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The measures for risk and uncertainty of ML applications can also become safety measures.

    “The measures for risk and uncertainty of ML applications can also become safety measures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Technical debts may result in maintenance cost escalation.

    “Technical debts which may result in maintenance cost escalation [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In the backward and forward snowballings, the additional papers to include were selected by one researcher.

    “In our backward and forward snowballings, the additional papers to include were selected by one researcher.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Interpretationcitation

    The same problem of lack of value focus may exist in ML research.

    “The same problem may exist in ML researches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Russell et al. discussed optimizing AI's economic impact, including labor market forecasting, market disruptions through AI techniques, and policy for managing adverse effects.

    “As another research priorities, Russell et al. [49] discussed optimizing AI's economic impact which includes labor market forecasting, market disruptions through the use of AI techniques and policy for managing adverse effects.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dahlmeier pointed out a lack of value focus in current NLP research.

    “They pointed out “lack of value focus” in the current NLP researches.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [84]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lucas et al. reported difficulties of translating ML results into real business impacts.

    “Lucas et al. [60] reported the difficulties of translating ML results into real business impacts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [60]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dahlmeier highlighted challenges that make it difficult to translate NLP research results into impactful innovation.

    “Dahlmeier [84] highlighted challenges that make it difficult to translate the results into impactful innovation in natural language processing (NLP) research.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [84]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The research method is based on existing systematic methods.

    “The research method designed for our purpose is based on the existing systematic methods [7,9].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The seven domain-specific papers mainly discussed domain-specific challenges; their challenges and solutions may be potentially related to software engineering activities for such domain applications.

    “They mainly discussed on domain specific challenges; their challenges and solutions may be potentially related to software engineering activities for such domain applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The relation mapping was conducted by the same researcher.

    “The relation mapping was also conducted by the same researcher.”
    Quote in the paperNot a substantial claim
69 of 69

  • The authors state this as an assumption.

    SE challenges for ML applications cover a similarly wide range of topics.

    “Given the various challenges in software engineering of ML, we surmise that SE challenges for ML applications cover a similarly wide range of topics.”

  • The authors state this as an assumption.

    The Swebok framework is helpful to seek answers to the research questions.

    “We presume that this comprehensive framework is helpful to seek answers to the following research questions.”

  • The paper relies on this without support.

    Most performance metrics on ML techniques are not easily understood by customers, who are expected to be unfamiliar with ML techniques.

    “Most of the performance metrics on ML techniques are not easily understood by customers, who are expected to be unfamiliar with ML techniques.”

  • The paper relies on this without support.

    The skills needed for ML application lifecycle works may go far beyond the scope of traditional software engineering.

    “The skills needed for these works may go far beyond the scope of traditional software engineering.”

  • The paper relies on this without support.

    Even the most comprehensive collection would provide only a snapshot because related papers are published daily.

    “Note that even the most comprehensive collection would provide only a snapshot because related papers are published daily.”

  • The paper relies on this without support.

    ML testing has drawn significant attention within the research and industrial communities because it is both important and difficult.

    “ML testing has drawn significant attention within the research and industrial communities because it is both important and difficult.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • [27] found that challenges in SE for ML are found in all KAs, and that safety/security and non-technical areas have received a lot of attention.

    “The author found that challenges in SE for ML are found in all KAs, and that safety/security and non-technical areas, in particular, have received a lot of attention in the literature.”

    From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [27]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Testing discussion is assigned to two different sections

    Minor

    The paper says the testing discussion is in Section 4.4, but later directs readers to Section 4.5 for that discussion. Section 4.5 covers software maintenance.

1 of 1

The citation lists haven’t been fetched yet

References 38 · Cited by 81

Authors
Kumeno, F.
Affiliations
Nippon Institute of Technology (JP)
Corresponding author
Kumeno, Fumihiro
Venue
Intelligent Decision Technologies
Publisher
SAGE Publications
Published
2020-02-10
Volume
13
Issue
4
Pages
463-476
ISSN
  • 1872-4981 (ISSN-L)
  • 1875-8843
DOI
10.3233/idt-190160
OpenAlex
W3006036236
Cited by
81
References
38
Citations by year
2026: 3 · 2025: 14 · 2024: 7 · 2023: 20 · 2022: 17 · 2021: 17 · 2020: 3
FWCI
13.42
Type
journal article
Language
en
Open access
Licence
Locations
Intelligent Decision Technologies (publishedVersion, cc-by-nc)
Funders
Japan Society for the Promotion of Science (19K03011)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
Keywords
machine learning applications, software engineering challenges, deep learning deployment, machine learning lifecycle, SWEBOK

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review