The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index; none of the 326 sources its claims cite is checked for support yet.
A software engineering perspective on engineering machine learning systems: State of the art and challenges
Open accessOverview
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic literature review synthesizes 141 SE-venue primary studies (2007–Dec 2020) on engineering machine learning systems. It finds that ML systems’ non-deterministic nature complicates all SE aspects, no SE aspect yet has mature tools and techniques, and testing is the most studied area but still lacks strong experimental proof.
Key findings
- The review analyzed 141 primary studies from SE venues; research interest increased sharply from 2018 onward, and 91% of studies appeared in conferences/workshops (129 papers) versus 12 in journals.
- Testing is by far the most popular area among researchers, but even for testing ML systems engineers have only tool prototypes and solution proposals with weak experimental proof; many challenges were identified through surveys and interviews.
- Among 93 experiment/case-study papers, 65 (70%) used only DL algorithms, 19 used traditional ML, 4 used both, and 5 did not report algorithms; image classification was the most popular application scenario, used in 57 studies.
- No SE aspect has a mature set of tools and techniques, and the paper calls for experiments and case studies, ideally in industrial environments, to better understand challenges and propose solutions.
Methods
- Systematic literature review following established guidelines, with database search (ACM, IEEE Xplore, ScienceDirect, Springer, Wiley, Google Scholar), backward/forward snowballing, and manual search of ICSE, ESEC/FSE, AIRE, and RAISE.
- Inclusion/exclusion criteria restricted papers to SE venues and to those involving at least one SE challenge of engineering ML systems; after de-duplication and quality assessment, the final pool was 141 studies.
- Quantitative analysis reported frequencies/percentages for research methods and application scenarios/datasets; qualitative open coding analyzed challenges and solutions, classified under SWEBOK-based SE knowledge areas.
Limitations
- Paper-stated: scope is limited to studies published until 31 December 2020, peer-reviewed academic venues (grey literature excluded), and SE venues only (AI/ML/DL/Data Science/Data Management venues excluded); primary study selection, data extraction, and synthesis are subject to researcher bias, which may be higher in a single-author SLR.
- Paper-stated: single data extraction can produce more errors than double data extraction, and the author notes that selection only from SE venues may miss relevant studies from other venues.
- Evident from text: the search used broad terms and a “search saturation effect,” checking only the first 200 results per query in each database unless results 190–200 still looked relevant, which may limit recall despite snowballing and manual search.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
225 Not checked yet · 51 Own result · 23 No basis given · 20 Own method · 14 Own interpretation · 2 Sources not found
In the paper
“Unlike Kumeno (2019), I was able to identify more papers published in SE venues (141 vs. 47) and provided a taxonomy of challenges and proposed solutions classified under SE's main knowledge areas.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno (2019)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Testing is by far the most popular area among researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of the challenges of ML systems engineering were identified through surveys and interviews.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite increasing interest from 2018 onwards, the results reveal that none of the SE aspects have a mature set of tools and techniques.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This review on the state-of-the-art is the most contemporary as it is the most up-to-date review.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even for testing ML systems, engineers have only some tool prototypes and solution proposals with weak experimental proof.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The included number of papers in Kumeno (2019), Lwakatare et al. (2020), Nascimento et al. (2020), and this study are 115, 72, 55, and 141, respectively.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno (2019)
- Not checked yetcites Lwakatare et al. (2020) source
- Not checked yetcites Nascimento et al. (2020)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Eight studies in our primary study pool (Haq et al., 2020;Harel-Canada et al., 2020;Jahangirova and Tonella, 2020;Li et al., 2019b;Stocco and Tonella, 2020;Stocco et al., 2020;Tian et al., 2018;Zhang et al., 2018a) conducted an experiment for predicting steering wheel angle.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Haq et al., 2020
- Not checked yetcites Harel-Canada et al., 2020
- Not checked yetcites Jahangirova and Tonella, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Image classification is the most popular application scenario used in 57 primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Twenty studies included a classification scenario other than image classification.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The third most popular application scenario (used in 13 studies) is autonomous driving, following image and other types of classification scenarios.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In line with these findings, DL algorithms were used in all primary studies involving autonomous driving, continuous value prediction, object detection, and mountain car problem scenarios.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One hundred twenty one out of 141 primary studies address only one knowledge area.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most of the empirical studies employed experiments.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Five studies addressed testing and maintenance aspects of ML systems by analyzing ML frameworks or libraries involved in these systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, nearly all of the image classification scenarios were also experimented using DL algorithms.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Four studies involved an application scenario to predict a numerical value for an observation.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For the classification scenarios excluding image classification, the researchers mostly used traditional ML algorithms.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The studies that used an experiment or a case study as a research method involved one or more application scenarios to observe SE challenges and validate a solution.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Based on the final paper pool, most challenges in developing ML systems manifest in issues related to data; models; dependencies; infrastructure and tools; ML algorithms, techniques, and libraries; and reuse.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Twenty-four studies include custom datasets constructed by researchers (including synthetic datasets) or privately owned by a company.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Recently, researchers started to explore conflicts between specific quality attributes, such as the trade-off between fairness and performance or accuracy (Kamishima et al., 2011).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with new types of conflicts between requirements:
What the sources say
- Not checked yetcites Kamishima et al., 2011
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Eleven studies used Udacity self-driving car challenge dataset for experiments in autonomous driving domain for various tasks such as predicting steering wheel angle (Kim et al., 2019;Li et al., 2019b;Tian et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Kim et al., 2019
- Not checked yetcites Li et al., 2019b
- Not checked yetcites Tian et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Twentynine datasets were only used once in an experiment or case study.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Since most of the studies are focused on image classification problems, most of the datasets in Table 7 are the datasets for image classification, such as MNIST (LeCun et al., 1998), CIFAR-10 ( Krizhevsky and Hinton, 2009), ImageNet (Fei-Fei, 2010), Fashion-MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), and IMDB-WIKI dataset.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites LeCun et al., 1998 source
- Not checked yetcites Krizhevsky and Hinton, 2009
- Not checked yetcites Fei-Fei, 2010
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Although transfer learning is an attractive topic in ML, it has not been addressed by the studies in the final paper pool.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Developers usually reported that analyzing and understanding the structure and behavior of ML models (mainly neural networks) is very difficult (Amershi et al., 2019;Arpteg et al., 2018;Hill et al., 2016;Xie et al., 2019a;Zhang et al., 2019b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Arpteg et al., 2018
- Not checked yetcites Hill et al., 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The papers in my pool mainly identify three groups of challenges, which are interrelated:”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Testing & Quality is the most focused knowledge area by academia and industry.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seven papers in the final pool address the challenges in two main categories from an organizational perspective: Having the required skill sets and having a team working harmoniously.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no paper or one paper per year until 2017.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As pointed out under RQ1, experiments are mostly used for testing and quality aspects.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Use of various research methods for all SE aspects:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The answer to RQ2 indicates that researchers have mainly used classification problems for their experiments and case studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Use of various application scenarios:
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As Table 8 reveals, many challenges have not been addressed.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For researchers and academicians:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“There are a few studies targeting development and maintenance.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Use of various research methods for all SE aspects:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While there are some examples of using ML algorithms for project management (two examples in ML for SE by Nassif et al. (2016) and Pospieszny et al. (2018)), our primary study pool does not include any practices for managing ML systems development projects.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, even for the difficulties addressed, the proposed solutions are mostly conceptual or implemented as a prototype tool without industrial use.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For researchers and academicians:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we can see in Figure 10, this research topic is mostly dominated by academicians, although engineering ML systems is a highly practical area.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Industry-academia collaboration:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The paper pool does not include any study that uses experiment to address requirements-and design-related challenges.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Use of various research methods for all SE aspects:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Researchers also conducted experiments on development & tools and maintenance & configuration management knowledge areas.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dominant research method, i.e., experiment, is primarily used in studies addressing testing and quality.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result of thematic and statistical analysis, the researchers have identified some difficulties in development & tools, test & quality, and organizational topics.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After excluding one study (found via DB search), the final primary study pool included 141 individual results.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plainn = 141- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Five papers did not report the algorithms used.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among 93 papers, 65 papers (70%) used only DL algorithms.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ML systems engineering in real-world settings is challenging since it adds additional complexity to engineering”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Despite several promising examples, 47% of AI projects remain prototypes due to the lack of the tools to develop and maintain a production-grade AI system, according to Gartner Research (Gartner, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 1 INTRODUCTION
What the sources say
- Not checked yetcites Gartner, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“To the best of my knowledge, Kumeno (2019) published the first SLR addressing the challenges of engineering ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno (2019)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The non-deterministic nature of ML systems complicates all SE aspects of engineering ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“Hence, for scenarios where there is a big gap between the inputs (such as images) and the outputs (such as objects), DL models are able to perform much better than traditional ML models (Zhou et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Zhou et al., 2018 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, large labelled datasets are required to train DL models (Fink et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Fink et al., 2020 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“On the other hand, DL algorithms are capable of learning more correct features compared to human-designed ones (Zhou et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Zhou et al., 2018 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ML model development practice differs from traditional software development due to its data dependency, uncertainty, and experimentation requirements.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Discovering, accessing, collecting, cleaning, and transforming data is challenging and time-consuming (Amershi et al., 2019;Baier et al., 2019;Correia et al., 2020;Fredriksson et al., 2020;Fujii et al., 2020;Hill et al., 2016;Kim et al., 2017;Lwakatare et al., 2019;Sankaran et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Baier et al., 2019
- Not checked yetcites Correia et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Customers generally do not understand the difference between traditional software and ML system (Baier et al., 2019;Vogelsang and Borg, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Managing expectations of customers:
What the sources say
- Not checked yetcites Baier et al., 2019
- Not checked yetcites Vogelsang and Borg, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Engineers do not have a mature set of solution patterns for solving the issues posed by ML.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“ML systems pose new challenges for conflict resolution (Horkoff, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with new types of conflicts between requirements:
What the sources say
- Not checked yetcites Horkoff, 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“It may also be difficult to map these measures to business objectives (Baier et al., 2019;de Souza Nascimento et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Baier et al., 2019
- Not checked yetcites de Souza Nascimento et al., 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“On the other hand, customers face difficulties in selecting proper evaluation metrics and interpreting these for assessing the quality of an ML system (Vogelsang and Borg, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Managing expectations of customers:
What the sources say
- Not checked yetcites Vogelsang and Borg, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A degradation in how well an ML system meets users' expectations should be expected on production in time (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Requirements to be fulfilled by ML components are generally specified using quantitative measures (such as accuracy, precision, recall, F measure) (Kim et al., 2017;Vogelsang and Borg, 2019;Wan et al., 2019) that are new for many stakeholders.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Kim et al., 2017 source
- Not checked yetcites Vogelsang and Borg, 2019
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“This principle prevents the use of standard techniques (such as abstraction and information hiding) for reducing coupling (Wan et al., 2019), which is a fundamental design principle for traditional software (Sculley et al., 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Sculley et al., 2014
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The development of ML systems makes the RE process even more challenging (Chechik, 2019;Dalpiaz and Niu, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.1 Requirements Engineering
What the sources say
- Not checked yetcites Chechik, 2019
- Not checked yetcites Dalpiaz and Niu, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In some domains, such as software analytics, making models explainable to users is as important as making accurate decisions (Dam et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Dam et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In addition, data quality is an important aspect to be considered during RE since it directly affects the performance of an ML model (Challa et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with changing emphasis on different requirements:
What the sources say
- Not checked yetcites Challa et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Working with a large volume of data that requires a distributed system brings additional complexity to development (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“On the other hand, such integration may be challenging due to the different characteristics of ML and traditional software components (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Amershi et al., 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Developers work on extracting selecting informative features for models (Amershi et al., 2019;Hill et al., 2016;Kim et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Hill et al., 2016
- Not checked yetcites Kim et al., 2017 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Developing models may require many iterations and experimentation, including many feedback loops between sub-activities (Amershi et al., 2019;Pham et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Pham et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“This heterogeneity causes potential discrepancies among development, quality assurance, and production environments (Zhang et al., 2020a).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Zhang et al., 2020a
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Tripakis (2018) and Menzies (2019) points out the heterogeneity of DL platforms that leads to interoperability problems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Tripakis, 2018
- Not checked yetcites Menzies, 2019 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This difficulty may lead to problems in evaluating and debugging models.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Due to the highly iterative nature of ML model development, the infrastructure should enable experiment management (Arpteg et al., 2018;Gharibi et al., 2019;Lwakatare et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Arpteg et al., 2018
- Not checked yetcites Gharibi et al., 2019
- Not checked yetcites Lwakatare et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Testing ML systems poses challenges that arise from the fundamentally different nature and development of ML systems (as briefly explained in Section 2.2), compared to relatively more deterministic traditional software systems (Zhang et al., 2020c).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Zhang et al., 2020c
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Handling exceptions in data is a serious problem in developing ML models (Zhang et al., 2020a).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites Zhang et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Many researchers mentioned the difficulty of test case generation for ML systems (Ahmed et al., 2020;Chechik et al., 2019;Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Ahmed et al., 2020
- Not checked yetcites Chechik et al., 2019
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In such cases, the changes in the ML model may negatively affect the dependent components (Belani et al., 2019;Hill et al., 2016;Lwakatare et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Belani et al., 2019
- Not checked yetcites Hill et al., 2016
- Not checked yetcites Lwakatare et al., 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The outputs of an ML model may be used by many other components implicitly.
In the paper
“Although there are some tools for exploring such data dependencies (McMahan et al., 2013), these are not as capable and common as the static analysis tools for traditional software.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites McMahan et al., 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A heterogeneous tool set challenges engineers in many tasks, such as data processing, environment setup, and model deployment (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A frequently seen type of bug in ML systems is a”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites Islam, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Several ML models are vulnerable to adversarial examples, which are only slightly different from correctly classified examples drawn from the training data distribution (Szegedy et al., 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Szegedy et al., 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The presence of diverse and incompatible programming and data tools is a severe concern for engineering ML systems (Kim et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Kim et al., 2017 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In some cases, a lot of manual work may be needed for deployment (Baier et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Baier et al., 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“On the other hand, ML components generally depend on external data (Sculley et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Sculley et al., 2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“One reason for this is the prevalence of non-determinism in the training process, which leads to hard-toreproduce bugs (Wan et al., 2019;Zhang et al., 2018b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Zhang et al., 2018b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All of these approaches try to reduce input data labeling and testing effort.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Adversarial examples cause ML models to produce an error output with high confidence, even though their difference from natural inputs are subtle (Szegedy et al., 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Szegedy et al., 2014
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“On the other hand, some of the generated images (test inputs) were still unnatural and could not be recognized by humans (Zhang et al., 2020c).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Zhang et al., 2020c
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In some cases, collecting testing datasets may require manual labeling, which is labor-intensive (Li et al., 2019b;Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Li et al., 2019b
- Not checked yetcites Wan et al., 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“ML systems heavily depend on ML libraries, frameworks, and platforms.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Guo et al., 2019
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Fairness can be an essential quality attribute of ML systems (Corbett-Davies and Goel, 2018), as pointed out in Section 5.3.1.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Corbett-Davies and Goel, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Unlike traditional software, code coverage cannot be used for ML components since the decision logic of ML capability is obtained via training an ML model, not via explicit coding (Ishikawa and Yoshioka, 2019;Khomh et al., 2018;Kim et al., 2019;Ma et al., 2018b;Sekhon and Fleming, 2019;Sun et al., 2018;Zhang et al., 2020c).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Ishikawa and Yoshioka, 2019
- Not checked yetcites Khomh et al., 2018 source
- Not checked yetcites Kim et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The non-deterministic nature of ML systems makes test evaluation more challenging (Arpteg et al., 2018;Ishikawa, 2018;Ishikawa and Yoshioka, 2019;Nishi et al., 2018;Tao et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Evaluating test results:
What the sources say
- Not checked yetcites Arpteg et al., 2018
- Not checked yetcites Ishikawa, 2018
- Not checked yetcites Ishikawa and Yoshioka, 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The bugs in these components impair ML systems (Guo et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Guo et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Moreover, good performance during testing cannot guarantee the satisfactory performance of ML systems on production (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Evaluating test results:
What the sources say
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Access to high quality test data is a concern for developing ML systems (Arpteg et al., 2018;Guo et al., 2018;Knauss et al., 2017;Li et al., 2019b;Ma et al., 2018a;Ma et al., 2018b;Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Arpteg et al., 2018
- Not checked yetcites Guo et al., 2018
- Not checked yetcites Knauss et al., 2017
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Unlike testing traditional software, finding one or a few incorrect results does not necessarily indicate the presence of a bug (Dwarakanath et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Evaluating test results:
What the sources say
- Not checked yetcites Dwarakanath et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Garcia et al. (2020) identified 499 bugs in these two software systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Garcia et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Test case evaluation is a complex task since it is hard to formalize and measure test cases' characteristics influence quality (Durelli et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Durelli et al., 2019 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The difficulties of debugging ML systems were pointed out by several studies (Khomh et al., 2018;Wan et al., 2019;Zhang et al., 2018b;Zhang et al., 2019a).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Khomh et al., 2018 source
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Zhang et al., 2018b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Another pain point is the lack of debugging and profiling support for ML systems (Zhang et al., 2019a;Zhang et al., 2020a).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Zhang et al., 2019a
- Not checked yetcites Zhang et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Data lifecycle activities form a critical and compelling subgroup of activities, which profoundly affect ML projects' success (Munappy et al., 2020;Polyzotis et al., 2018;Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Munappy et al., 2020
- Not checked yetcites Polyzotis et al., 2018 source
- Not checked yetcites Wan et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Software Composition Analysis (SCA) tools can help in discovering all related components, their supporting libraries, and their direct and indirect dependencies (Han et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Han et al., 2020
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Another critical maintenance issue is the technical debts present in ML frameworks (Liu et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Liu et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Automatic maintenance mechanisms can be built to keep the performance at a certain level (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Wan et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Liu et al. (2020b) identified design, defect, documentation, test, requirement, compatibility, and algorithm debts in ML frameworks, which may cause maintenance problems in ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Liu et al., 2020b
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Unforeseen changes in the external world may cause changes in input data patterns and negatively affect the performance of ML components (known as “concept drift” (Tsymbal, 2004), see Section 5.3.2).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Tsymbal, 2004
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Data and ML models are essential configuration items besides code (Belani et al., 2019;Sato et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Belani et al., 2019
- Not checked yetcites Sato et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The development of ML components is an iterative and experimental process (Gharibi et al., 2019), and hence, it is prevalent to perform a vast number of experiments to identify the optimal ML model (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Maintaining ML systems involves keeping track of additional configuration items, models, and data, besides code (Sato et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Sato et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The development of ML components is an iterative and experimental process (Gharibi et al., 2019), and hence, it is prevalent to perform a vast number of experiments to identify the optimal ML model (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Gharibi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“( 2) some experiments may be repeated unintentionally (Hill et al., 2016);”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Hill et al., 2016
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“It is a challenge to keep pace with the latest versions of ML libraries and their dependencies (Han et al., 2020).”
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
“Keeping pace with the latest versions of ML libraries and their dependencies is a challenge (Han et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Han et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: ML models are dependent on ML frameworks.
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
None of the sources cited for this claim have been checked yet.
In the paper
“ML model lifecycle activities are another vital subgroup of activities integrated with the SE process (Alvarez-Rodríguez et al., 2019;Byrne, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Alvarez-Rodríguez et al., 2019
- Not checked yetcites Byrne, 2017
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“(1) experiments may not be reproduced when needed (Hill et al., 2016;Zaharia et al., 2018);”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Hill et al., 2016
- Not checked yetcites Zaharia et al., 2018
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Model management, which refers to tracking, storing, query, comparing, reproducing, and sharing models (Gharibi et al., 2019), has been identified as one of the most challenging and time-consuming activities (Schelter et al., 2018;Vartak, 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Schelter et al., 2018
- Not checked yetcites Vartak, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Teams may require to compare, reproduce, and share experiments (Gharibi et al., 2019;Lwakatare et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Gharibi et al., 2019
- Not checked yetcites Lwakatare et al., 2019
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Another challenge arises when organizations decide to enhance their existing systems with ML capabilities (Jahić and Roitsch, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Jahić and Roitsch, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This trend clearly shows the increasing interest of the SE community in engineering ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Part of the organizational challenges involves identifying skill sets and roles required to engineer ML systems (Flaounas, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Flaounas, 2017
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“As a result of not being able to estimate effort, business owners act impatiently and cancel ML component development despite promising intermediate results (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Estimating effort:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“It is even challenging to have all these people on the same page (Kim et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Kim et al., 2017 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Lack of developer support tools for data-intensive systems is a challenge for developing ML systems (Anderson, 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Anderson, 2015
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Engineering an ML system involves skill sets from multiple fields, i.e., machine learning, deep learning, data science, mathematics, algorithms, software engineering, and statistics (Alshangiti et al., 2019;Alvarez-Rodríguez et al., 2019;Belani et al., 2019;Correia et al., 2020;Wan et al., 2019;Workera, 2020).”
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
“Engineering an ML system involves skill sets from various fields, i.e., machine learning, deep learning, data science, mathematics, algorithms, software engineering, and statistics (Alshangiti et al., 2019;Alvarez-Rodríguez et al., 2019;Belani et al., 2019;Wan et al., 2019;Workera, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
- Not checked yetcites Alvarez-Rodríguez et al., 2019
- Not checked yetcites Belani et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
In the paper
“In some cases, the team cannot know whether the quantitative targets are achievable until final ML models are obtained (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Estimating effort:
What the sources say
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For instance, the synergy between data scientists, domain experts, and customers during data processing is essential for proper data collection, data enrichment and labeling (Correia et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Correia et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Some tools emerged from academia, such as DARVIZ (Sankaran et al., 2017), DeepVisual (Xie et al., 2019a), NeuralVis (Zhang et al., 2019b), ModelKB (Gharibi et al., 2019), to solve specific problems in developing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Sankaran et al., 2017
- Not checked yetcites Xie et al., 2019a
- Not checked yetcites Zhang et al., 2019b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“A proliferation of roles, including specialists, in particular, SE phases, such as design (Northrop et al., 2019) or specific types of algorithms such as natural language processing (Byrne, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Northrop et al., 2019 source
- Not checked yetcites Byrne, 2017
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Practitioners and academicians already started to identify emerging roles to classify skill sets and have proper people in teams to engineer ML systems (Kim et al., 2017;Workera, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Kim et al., 2017 source
- Not checked yetcites Workera, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“It is difficult and mostly impossible to characterize all the behaviors of ML components under all circumstances (Tuncali et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Tuncali et al., 2019 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Therefore, it is challenging to get all these people working together towards achieving a common objective (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“To support other SE aspects, such as generating Python components ready for deployment, version control conflict handling, and continuous integration, some tools, such as nbdev, have been released (Howard and Gugger, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Howard and Gugger, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite many studies, we do not have mature testing techniques and tools to develop ML systems in industrial settings.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“There are some platforms, such as MLflow (Zaharia et al., 2018), focusing only on the ML lifecycle.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Zaharia et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Open-source ML libraries include many bugs (Garcia et al., Pham et al., 2019;2020;Wang et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Garcia et al.
- Not checked yetcites Pham et al., 2019
- Not checked yetcites Wang et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, selection of primary studies only from SE venues forms a threat to having a complete set of primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Debugging and profiling tool support is an essential deficiency for engineering ML systems (Zhang et al., 2019a).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Zhang et al., 2019a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Coverage criteria designed for ML/DL models have been used to evaluate test cases (Ma et al., 2018a;Sun et al., 2018;Sun et al., 2019b;Du et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Ma et al., 2018a
- Not checked yetcites Sun et al., 2018
- Not checked yetcites Sun et al., 2019b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, compared to tool support for traditional software development, there is room for improvement to have SE tooling for developing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Mutation testing (MuT) has been used to address the shortcomings of coverage criteria (Jia and Harman, 2010;Zhang et al., 2018c).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Jia and Harman, 2010
- Not checked yetcites Zhang et al., 2018c
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Another maintainability issue related to ML libraries is the technical debt present in these libraries (Liu et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Liu et al., 2020b
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“As in testing scientific software (Kanewala and Bieman, 2014), perfect oracles may not be available for ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Kanewala and Bieman, 2014 source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For instance, there is empirical evidence showing that single data extraction results in more errors than double data extraction (Buscemi et al., 2006).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
What the sources say
- Not checked yetcites Buscemi et al., 2006 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 62
- Probably not a substantial claim. 155
- The quote only partly states this claim. 1
In the paper
“While none of the existing reviews addressing SE aspects (Kumeno, 2019;Lwakatare et al., 2020;Nascimento et al., 2020) cover the papers published in 2020, this study covers the papers published until the end of 2020 in SE venues (75 papers in total).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno, 2019;Lwakatare et al., 2020;Nascimento et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to these three primary studies using multiple research methods, the total number of primary studies equals 147 in Figure 6.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thematic analysis, case study, and statistical analysis are the other approaches used by the researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Four studies mention ML systems predicting values for combined sewer overflow (Challa et al., 2020), stock price (Wang et al., 2020b), age (Gao et al., 2020a), real estate value and weather (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Challa et al., 2020
- Not checked yetcites Wang et al., 2020b
- Not checked yetcites Gao et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One case study (Nakamichi et al., 2020) addressing requirements engineering challenges did not report its application scenario(s).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Nakamichi et al., 2020
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 8 shows the distribution of application scenarios used in 93 of the primary studies, the research method of which is either an experiment or a case study.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“One of their experiments involved BERT (Devlin et al., 2018) model over GLUE (General Language Understanding Evaluation) benchmark (Wang et al., 2018), with various batch sizes and sequence lengths.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Devlin et al., 2018
- Not checked yetcites Wang et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Liu et al. (2020b) explored technical debts in seven open-source DL frameworks and their impacts on the maintainability of DL systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Liu et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Three studies involved an ML system for object detection.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides experiments, the researchers interviewed and surveyed experts to learn about the challenges they face in engineering ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Tizpaz-Niari et al. (2020) discovered performance bugs in ML frameworks, such as the implementation of logistic regression in scikit-learn.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Tizpaz-Niari et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Only one study (Murphy et al., 2007) is concerned with a ranking problem.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Two other studies (Humbatova et al., 2020;Zhang et al., 2020a) used thematic analysis and interview methods together.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Humbatova et al., 2020
- Not checked yetcites Zhang et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Sun and Zhou (2018) and Zheng et al. (2019) addressed the challenges of testing an ML system on machine translation.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Sun and Zhou, 2018
- Not checked yetcites Zheng et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There are also many terms representing the intervention of this study, i.e.,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“I divided the included studies into empirical (127 papers) and non-empirical (15 papers), as suggested by Mohanani et al. (2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
What the sources say
- Not checked yetcites Mohanani et al. (2018)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Ahmed et al. (2020), Dutta et al. (2020), and Wang et al. (2020b) studied problems in DL/ML frameworks related to testing and quality.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Ahmed et al., 2020
- Not checked yetcites Dutta et al., 2020
- Not checked yetcites Wang et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Two studies (Amershi et al., 2019;Wan et al., 2019) combined interview and survey methods.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can observe many terms that represent the population of this study, i.e.,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Alshangiti et al. (2019) performed thematic and statistical analysis together.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Murphy et al. (2007) aim at ranking the probabilities of observing a failure in a device.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Pham et al. (2020) conducted experiments to study the variance of DL systems and a survey to understand the awareness of this variance among researchers and practitioners.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Pham et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three studies contained application scenarios of Natural Language Processing, two specifically machine translation.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fifteen papers either provide the author's opinions or did not report enough details about the research method.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Trujillo et al. (2020) used mountain car problem to investigate whether neuron coverage can be used for deep reinforcement learning systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Trujillo et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 7 shows the list of datasets used more than once in the experiments and case studies in 93 primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Gao et al. (2020b) proposed an estimation tool for GPU memory consumption of DL models to help developers in planning runtime resource requirements.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Gao et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally yet importantly,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They reported one of the complexities posed by ML systems as the”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Sculley et al., 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Washizaki et al. (2019b) state that ad-hoc solutions are being used for solving everyday problems in developing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Washizaki et al., 2019b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Wan et al. (2019) interviewed professionals with experience in both traditional and ML system development and reported that nearly every interviewee made a strong statement about differences between the requirements of ML versus non-ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.1 Requirements Engineering
What the sources say
- Not checked yetcites Wan et al., 2019
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Benton et al. (2020) proposes an architecture for an ML system that deals with high-volume data.This architecture includes a component for data federation to deal with structured, unstructured, and streaming data.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Benton et al., 2020 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For instance, Jiarpakdee et al. (2020) argued that the lack of explainability of defect prediction models could hinder the adoption of these models in practice.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Jiarpakdee et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“184 datasets were reported in 93 studies, which means most of the studies used two datasets on average.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“John et al. (2020a; 2020b) proposed a framework involving five architectural alternatives ranging from centralized cloud to fully decentralized edge architectures for designing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites John et al., 2020a
- Not checked yetcites John et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Yokoyama (2019) proposes an architectural pattern for locating performance problems and rolling back in case of a failure on production.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Yokoyama, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Washizaki et al. (2020) identified 15 design patterns for ML systems and plan to expand the list via further empirical studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Washizaki et al., 2020
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“To provide more comprehensive knowledge at different level of expertise, Hashemi et al. (2020) automatically generated ML system documentation from StackOverflow Q&As.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Hashemi et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“I classified these challenges into eight categories.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, the top SE conferences, i.e., ESEC/FSE and ICSE, receive submissions on this emerging topic.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The number of primary studies in SE venues on engineering ML systems has approximately doubled annually every year since 2018.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One hundred and one papers (72%) were authored solely by academic researchers, 11 papers (8%) by researchers from research organizations or companies, and 29 papers (20%) were written jointly by universities and other organizations.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Development & tools and requirements engineering areas the second and third areas in which researchers collaborated.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three papers were published in 2017, after which we see a sharp increase starting from 2018.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, researchers collaborated the most within the scope of testing and quality.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The earliest paper (Murphy et al., 2007) was published at the International Conference on Software Engineering & Knowledge Engineering in 2007.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Regression testing of ML systems has not been addressed in the primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“SE researchers focused mostly on testing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Planning, controlling, and monitoring of testing activities for ML systems have not been addressed by researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The conference with the most papers (20) is the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Researchers have been showing an increasing interest in this research area since 2018 (Figure 9).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 7 CONCLUSIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Case studies enabled researchers to collect data on activities related to ML systems in the industry.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, experts mentioned many challenges covering all aspects of engineering ML systems in interviews and surveys.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure4, the DB search yielded 90 results, snowballing 68, and manual search 58.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plainn = 141- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The journals with the most papers (three in each) are IEEE Access, IEEE Software, and IEEE Transactions on Software Engineering.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“91% of the studies were presented in conferences and workshops (129 papers in conferences/workshops and 12 papers in journals).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nineteen papers used traditional ML algorithms, and four papers used both DL and traditional ML algorithms.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although there are a lot of primary studies (Section 9.1) and secondary studies (Section 3) on this topic, many research questions remain unanswered (Section 6.1 and Table 8).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 7 CONCLUSIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were 77 studies found via only one search method (DB search: 40, Snowballing: 31, Manual search: 6) and 64 studies via more than search method.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plainn = 141- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Many researchers from software engineering (SE) (Hesenius et al., 2019;Northrop et al., 2019) and ML (Karpathy, 2017;Menzies, 2019), as well as industry practitioners (Benton, 2020;Booch, 2019;Lin and Kolcz, 2012) have stated the requirement of such a holistic view.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 1 INTRODUCTION
What the sources say
- Not matchedcites Hesenius et al., 2019;Northrop et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Karpathy, 2017;Menzies, 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Benton, 2020;Booch, 2019;Lin and Kolcz, 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“The research method was based on established guidelines (Kitchenham and Charters, 2007;Wohlin, 2014), some previous good examples (Durelli et al., 2019;Mohanani et al., 2018), and my previous experience in conducting SLR (Garousi et al., 2019;Giray and Tüzün, 2018;Tarhan and Giray, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4 RESEARCH OBJECTIVES AND METHOD
What the sources say
- Not matchedcites Kitchenham and Charters, 2007;Wohlin, 2014
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Durelli et al., 2019;Mohanani et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Garousi et al., 2019;Giray and Tüzün, 2018;Tarhan and Giray, 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To maintain a high-quality input of primary studies for this SLR, I decided to exclude the papers with a score lower than six points out of 12 for empirical studies and three points out of six for non-empirical studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“To assess the quality of primary studies, I used the quality assessment criteria (Table 5) proposed by Mohanani et al. (2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
What the sources say
- Not checked yetcites Mohanani et al. (2018)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all primary studies, I recorded the challenges and solutions to the spreadsheet without making any changes as written in the primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“I used the knowledge areas presented in Wan et al. (2019), which was derived from SWEBOK (Bourque and Fairley, 2014) to classify the challenges and solutions.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
What the sources say
- Not checked yetcites Wan et al. (2019)
- Not checked yetcites Bourque and Fairley, 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“As briefly explained in Section 2.2, traditional ML algorithms entail a manual feature selection process, which may require domain expertise, statistical analysis of input data, and experiments for building models with different feature sets (Kelleher, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Kelleher, 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“As these examples show, the industry is calling for action to resolve the challenges of engineering ML systems and to propose new techniques to cope with the additional complexity of ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > SE
What the sources say
- Not checked yetcites Lorica and Loukides (2018)
- Not checked yetcites Heck, 2019
- Not checked yetcites Sato et al. (2019)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“showing researchers' high interest in testing ML systems”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“I conducted open coding (Miles and Huberman, 1994) to analyze the challenges and solutions.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.4 DATA SYNTHESIS AND REPORTING
What the sources say
- Not checked yetcites Miles and Huberman, 1994
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I excluded one study with a score under the threshold.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This change poses two crucial challenges: detecting when concept drift occurs and keeping the models up-to-date (Maimon and Rokach, 2010).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Maimon and Rokach, 2010
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“There is a need for a catalog of patterns dedicated to ML-specific problems (Washizaki et al., 2019b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Washizaki et al., 2019b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Requirements of ML systems are heavily dependent on available data; different data sets may lead to different needs (Rahimi et al., 2019;Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Eliciting and analyzing requirements:
What the sources say
- Not checked yetcites Rahimi et al., 2019
- Not checked yetcites Wan et al., 2019
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Requirements are much more uncertain (Belani et al., 2019;Chechik, 2019;Ishikawa and Yoshioka, 2019;Rahimi et al., 2019;Wan et al., 2019), and hence they are more difficult to elicit and analyze.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Eliciting and analyzing requirements:
What the sources say
- Not checked yetcites Belani et al., 2019
- Not checked yetcites Chechik, 2019
- Not checked yetcites Ishikawa and Yoshioka, 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Explainability is quite essential in building trust between an ML system and its users (Nguyen-Duc et al., 2020;Ribeiro et al., 2016).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Nguyen-Duc et al., 2020
- Not checked yetcites Ribeiro et al., 2016
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition, combining data is not always straightforward (Kim et al., 2017;Lwakatare et al., 2019) since different systems can have other objectives for storing data, hence leading to data semantics heterogeneity and data integration problems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites Kim et al., 2017 source
- Not checked yetcites Lwakatare et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Distributed architectural patterns are widely used in designing ML systems to cope with high-volume data (Wan et al., 2019), which leads to additional complexity in architectural and detailed design (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Wan et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Known as”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Tsymbal, 2004
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Trade-offs among device cost, ML model performance, and data privacy requirements guide the selection of an architectural alternative (John et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites John et al., 2020b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They may need to optimize a possible tradeoff between the best features and model complexity (Sapp, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Sapp, 2017
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although non-testability is attributed to a lack of reliable test oracle (Weyuker, 1982), there are several challenges in testing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Weyuker, 1982 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Model deployment involves some challenges (Chen et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Chen et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Adapting the implementation of a neural network for a different task (Zhang et al., 2019a) or transferring a built solution to another domain (Baier et al., 2019) remains a challenge.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Zhang et al., 2019a
- Not checked yetcites Baier et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Another essential concern in developing ML models is fairness (Corbett-Davies and Goel, 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Corbett-Davies and Goel, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even the model requirements may not be apparent before starting with model development and may be clarified after many experiment iterations.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“They face challenges in understanding these (Alshangiti et al., 2019;Zhang et al., 2019a;Zhang et al., 2020a), especially the ones related to DL (Alshangiti et al., 2019), and identify a proper model (Islam, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Alshangiti et al., 2019
- Not checked yetcites Zhang et al., 2019a
- Not checked yetcites Zhang et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This development endeavor involves some sub-activities, such as feature engineering, model training, model evaluation, and model deployment (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In 2007, Murphy et al. (2007) mentioned the idea of testing ML systems and classified such systems as”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Zhou et al. (2020b) found out that OoD-aware ML models are more robust according to their experiments.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Zhou et al., 2020b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While reusing ML libraries is very common, effective, and efficient, reusing ML models in different domains or systems is not straightforward (Alvarez-Rodríguez et al., 2019;Amershi et al., 2019;Baier et al., 2019;Guo et al., 2019;Sankaran et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Alvarez-Rodríguez et al., 2019
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Baier et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For DL models, feature engineering is performed implicitly during model training (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Distributed systems require further knowledge and experience and additional cost and management of associated hardware and software (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For instance, imbalanced datasets will most likely lead to inaccurate prediction models (He and Garcia, 2009).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites He and Garcia, 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Welldesigned test cases can help in finding many types of defects in ML systems (Humbatova et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Humbatova et al., 2020
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Nevertheless, DL models should be evaluated using various important metrics such as robustness and neuron coverage, besides the widely-used prediction accuracy metric (Tian et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Tian et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Galhotra et al. (2017) proposed a technique to measure causal discrimination and managed to detect significant ML model discrimination towards gender, marital status, or race as many as 77.2% of the individuals in datasets.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Galhotra et al., 2017
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For instance, according to the experiments conducted by Haq et al. (2020), simulator-generated datasets yielded DNN prediction errors that are similar to those obtained via real-world datasets in autonomous driving domain.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Haq et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The discovery of adversarial examples and evaluation of model robustness are fundamental problems in engineering ML systems (Du et al., 2020;Fujii et al., 2020;Sun et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Du et al., 2020
- Not checked yetcites Fujii et al., 2020
- Not checked yetcites Sun et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Zhou et al. (2020a) presented their approach named DeepReduce, which helped them reduce testing data 7.5% on average and still reliably estimate the performance of DL models.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Zhou et al., 2020a
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Based on their preliminary results, neuron coverage is not sufficient to reach reliable conclusions about the quality DeepRL models (Trujillo et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Trujillo et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Yan et al. (2020) claimed that DNN coverage criteria do not have monotonic relations with model quality measured by model accuracy in the presence of adversarial examples.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Yan et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They classified these bugs into 13 root causes, 20 bug symptoms, and 18 categories of software components these bugs often affect (Garcia et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Garcia et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Mutation Testing (MuT) is a fault-based testing technique that claims to address the shortcomings of coverage criteria (Jia and Harman, 2010;Zhang et al., 2018c).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Jia and Harman, 2010
- Not checked yetcites Zhang et al., 2018c
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Based on the experiments conducted by Harel-Canada et al. (2020), higher neuron coverage led to fewer defects detected, less natural inputs, and more biased prediction preferences.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Harel-Canada et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Kim et al. (2019) and Sekhon and Fleming (2019) state that existing coverage criteria are not sufficiently fine-grained to capture subtle behaviors exhibited by ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Kim et al., 2019
- Not checked yetcites Sekhon and Fleming, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although researchers try to develop new coverage criteria appropriate for DL models, Li et al. (2019) could not find a strong correlation between the number of misclassified inputs in a test set and its structural coverage.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Li et al., 2019
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The size of test case sets is enormous, and missing test cases is a more challenging problem for ML systems (Khomh et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Khomh et al., 2018 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Flaky tests are observed frequently and may cause problems in software development process (Luo et al., 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Luo et al., 2014
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Zhou et al., 2015 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Failing to update the code logic accordingly as ML models evolve may introduce bugs and fail to achieve the optimal performance of ML models, and further negatively influence the entire ML-powered system (Peng et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Peng et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such mechanisms can detect performance degradation and take the required actions by re-training ML models with new data.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For instance, Hulten (2018) breaks down the ML process into five stages: (1) getting data to model, (2) feature engineering, (3) modeling, (4) deployment, and (5) maintenance.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Hulten, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Shams (2018) demonstrates a threephase ML product development process to shorten the delivery time from idea generation to deployment.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Shams, 2018 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Canary release approach may limit the possible negative impact of the new ML model and help to cope with the risk of deploying an updated ML model (Yokoyama, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Yokoyama, 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“These components can include hardware (GPU models primarily), platform (operating system and installed packages), source code (model training and pre-processing), configuration (model configuration and pre-processing settings), training data (input signals and target values), and model state (versions of trained models) (Arpteg et al., 2018;Wan et al., 2019;Yokoyama, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Arpteg et al., 2018
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Yokoyama, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Each experiment may have many components that affect the outcome, i.e., the ML model.
In the paper
“They found that developers are mostly fixing data dimensions and neural network connectivity (Islam et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Islam et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Rao (2019) shows the typical steps involved in the model development lifecycle: (1) define the problem, (2) collect, cleanse, and prepare data, (3) build and train model, (4) tune hyperparameters and validate model, and (5) deploy to production.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Rao, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Therefore, experiment management is a recent and complicated challenge in engineering ML systems (Hazelwood et al., 2018;Zaharia et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Hazelwood et al., 2018
- Not checked yetcites Zaharia et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Some large-scale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.
- Rests on an assumption: Each experiment may have many components that affect the outcome, i.e., the ML model.
- Rests on an assumption: It is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.
In the paper
“While versioning data is a vital need (Alvarez-Rodríguez et al., 2019;Amershi et al., 2019;Belani et al., 2019;Wan et al., 2019;Yokoyama, 2019), the methods and tools to support this is not mature yet (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Peng et al. (2020) observed that as ML models evolve, it poses maintenance challenges on the code logic that should coevolve.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Peng et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For instance, Kubernetes ecosystem may ease maintenance and problem localization (Benton, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Benton, 2020 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“While developers fix bugs, they may introduce adversarial vulnerabilities and new bugs (Islam et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Islam et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) experiments and their results may not be analyzed, reported, and shared appropriately.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The quote only partly states this claim.
In the paper
“Whether done automatically or manually, experimentation activities generate many artifacts (datasets, hyperparameters, models, libraries, etc.), which should be versioned (Golovin et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Golovin et al., 2017
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“In risky domains, automated checks can be used to trigger notifications if predefined thresholds are violated (Baier et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Baier et al., 2019
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Versioning data does not only involve keeping track of datasets but also keeping track of metadata of datasets (Gebru et al., 2018) and their relationships to ML models (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Gebru et al., 2018
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The steps are not executed as a waterfall process; there is feedback from deployment to problem definition and an iteration from building a model to deployment (Rao, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Rao, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“In another study, Pan (2020) observed that bug fixes may cause both an increase and decrease in robustness of DL models.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Pan, 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Another option is to augment training data with the guidance of newly identified input data and retrain ML model (Ren et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Ren et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“With the emergence of many general-purpose ML libraries, services, platforms combined with the accumulation of ML experience, developing ML systems has become relatively easy and cheap, whereas maintaining them over time is difficult and expensive (Sculley et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Sculley et al., 2015
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Newly identified data can be used directly for adaptive retraining (Stocco and Tonella, 2020).”
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
“The identified data can be used for adaptive retraining.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Stocco and Tonella, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
In the paper
“An ML engineer need to understand the mathematical and statistical background of models for proper model fitting and tuning (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Moreb et al. (2020) proposed a process for engineering ML systems in health informatics domain.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Moreb et al., 2020 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A software engineer may have to deal with distributed systems, web services, and multi-threaded programming (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Due to the uncertainties in the ML process, it is hard to estimate effort and do planning (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Estimating effort:
What the sources say
- Not checked yetcites Wan et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The popularity of ML has brought many tools, mostly open-source (Louridas and Ebert, 2016).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Louridas and Ebert, 2016
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“According to Sensemaking-Coevolution-Implementation Theory, sensemaking is one of the core activities in engineering software systems (Ralph, 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Ralph, 2015 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For example, while data scientists may be pragmatic when coding as long as they achieve the desired results, software engineers may care about code quality and maintainability (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Arpteg et al., 2018
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Components with ML capabilities are becoming an architectural part of software systems, sharing cross-cutting functional and non-functional concerns (Nguyen-Duc et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Nguyen-Duc et al., 2020
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“People having different backgrounds have cultural (Arpteg et al., 2018) and language differences.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Engineering complex systems, such as ML systems, is a complicated endeavor with organizational and technical aspects (Clarke et al., 2016).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Clarke et al., 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Wolf and Paine (2020) emphasizes the difficulty of sensemaking for business context, ML algorithms, libraries, and models in ML systems engineering projects.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Wolf and Paine, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Alvarez-Rodríguez et al. (2019) identified the formation of a workforce equipped with the required skills to tackle different types of problems in various domains as a challenge.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alvarez-Rodríguez et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A data preparer requires a good data slicing and dicing skills for proper data cleaning and feature preprocessing (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Monitoring for potential performance degradation on production (Wan et al., 2019;Yokoyama, 2019) and high-volume data processing (de Souza Nascimento et al., 2019;Wan et al., 2019) are two important design considerations for ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Yokoyama, 2019
- Not checked yetcites de Souza Nascimento et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reason for having much fewer papers in journals may be due to the fact that this topic is still in its infancy.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“There are some early attempts to determine the ML process' maturity (Amershi et al., 2019) and how ML components are used in larger software systems (Lwakatare et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Lwakatare et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“There are some efforts to bring various tools together to support the effective development of ML systems (Smith et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Smith et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“However, this diverse set of tools have emerged with challenges, such as compatibility (Rahman et al., 2019), integration of various components (Rahman et al., 2019), required knowledge and experience, etc.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rahman et al., 2019
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“ML systems' development involves a diverse set of frameworks, tools, libraries, and programming languages (Rahman et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rahman et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“While there are generic roles such as data scientist, ML engineer, data analyst, ML software engineer, ML researcher, and software engineer, there are more specific roles, such as data evangelist, data preparer, data shaper, data analyzer (Kim et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Kim et al., 2017 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I used broad terms to form search keywords to reduce the risk of excluding potentially relevant studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Date: This study covers primary studies published until the end of 2020, i.e., 31 December 2020.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“However, the success of these coverage criteria is questionable (Li et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Li et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“To maintain the performance of an ML system above a level that meets customer expectations, drift detectors may be developed to identify data deviating from training data distribution (Stocco and Tonella, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Stocco and Tonella, 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A set of guidelines was published to ensure ML systems' quality (QA4AI, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites QA4AI, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Perspective: The primary studies have been selected from SE venues to reflect an SE perspective on ML systems engineering.Therefore, papers published in AI, ML, DL, Data Science, Data Management venues have been excluded.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“With this approach, I decreased precision and increased recall found more candidate papers to be assessed for specifying the final set of primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“It discusses how to take an idea and an ML model developed by data scientists and deploy it as part of a scalable and maintainable system.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites Kästner and Kang, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Rudraraju and Boyanapally, (2019) focused on developing a data quality model for ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rudraraju and Boyanapally, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Personal bias might have been introduced during the application of inclusion and exclusion criteria.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They identified new data quality attributes that are not present in the Data Quality model defined in the standard ISO/IEC 25012.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rudraraju and Boyanapally, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“As an example of the fulfillment of this need, the QA4AI Consortium was established in 2018 in Japan to discuss the quality assurance of ML systems (Fujii et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Fujii et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of the concerns, such as some non-functional requirements, may be domain-specific (Nguyen-Duc et al., 2020) and can be discovered by exploring various ML problems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Use of various application scenarios:
What the sources say
- Not checked yetcites Nguyen-Duc et al., 2020
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The 2020 version of the Future of Jobs Survey reveals that the top increasingly strategic job roles are data analysts and scientists, AI and ML specialists, robotics engineers, software and application developers, and digital transformation specialists (World Economic Forum, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites World Economic Forum, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Type of Literature: This study comprises studies published in peer-reviewed academic venues.Grey literature, e.g., papers only published in arxiv.org,blogs, videos, etc., have been excluded.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate the risk of missing some relevant studies, I also carried out both backward and forward snowballing along with a manual search.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Tools to estimate computation and memory requirements, such as DNNMem (Gao et al., 2020b), may help developers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Gao et al., 2020b
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I also searched for five widely used online databases in SMS and SLR studies in computer science and software engineering.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Researchers analyzed and classified these bugs (Garcia et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Garcia et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One of the pioneer courses is the”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites Kästner and Kang, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Automating tests has become a necessity to cope with large-scale complex systems (Mosley and Posey, 2002).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Mosley and Posey, 2002
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While there are many processes for ML lifecycle (Hulten, 2018;Rao, 2019;Shams, 2018) and traditional software development (Klünder et al., 2019), a set of harmonized practices for developing ML systems is required (Alvarez-Rodríguez et al., 2019;Belani et al., 2019;de Souza Nascimento et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Hulten, 2018
- Not checked yetcites Rao, 2019
- Not checked yetcites Shams, 2018 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Wan et al.only mentioned the difficulty of effort estimation in ML projects (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While there are some examples of using ML algorithms for project management (two examples in ML for SE by Nassif et al. (2016) and Pospieszny et al. (2018)), our primary study pool does not include any practices for managing ML systems development projects.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Validity considerations are applicable for SMS and SLR studies, like empirical studies (Petersen et al., 2008;Petersen et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
What the sources say
- Not checked yetcites Petersen et al., 2008
- Not checked yetcites Petersen et al., 2015 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There may be other relevant studies published in AI, ML, DL, Data Science, and Data Management venues.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One of the challenges to be addressed is the identification of metamorphic relations that should be satisfied by an ML system (Kanewala and Bieman, 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Kanewala and Bieman, 2014 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The course takes a SE perspective on building software systems with a significant AI/ML component.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites Kästner and Kang, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides these five databases, I also searched Google Scholar to enrich the paper pool.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More testing can be conducted via test automation to reduce the risks of ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Two promising techniques for designing test cases are metamorphic testing techniques and leveraging adversarial techniques to detect erroneous behavior (Mjeda and Botterweck, 2019) on which further research is needed.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Mjeda and Botterweck, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The threats to this SLR's validity are mainly related to the specification of the candidate pool of papers, primary study selection bias, data extraction, and data synthesis.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To minimize this type of bias and errors whenever I could not decide on inclusion/exclusion by reading the abstract, I scanned the full text for the final decision.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The validity of data extraction is another essential aspect that directly affects this study's results.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I applied the open coding technique iteratively and incrementally to identify the challenges and solutions (RQ3).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This coding process potentially entails some researcher bias.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, primary study selection, data extraction, and synthesis are subject to researcher bias, and this bias may be higher than those SLRs with multiple authors.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to ensure the accuracy of the extracted data, I used existing categories in the literature where possible (RQ1, RQ2, and SE knowledge areas in RQ3).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I aimed to decrease the risk of researcher bias by mapping the relevant data in primary studies to the specified groups.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Despite its disadvantages, we can find SLRs with single authors in SE (Malhotra, 2015) and other research areas (Mitchell, 2011).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
The search space can be restricted by assuming a search saturation effect, such that checking the first 200 results of each query in each database is sufficient.
“Therefore, as others have done (Adams et al., 2016;Garousi and Mäntylä, 2016;Garousi et al., 2018;Godin et al., 2015), I had to restrict the search space by assuming a”
The paper relies on this without support.
The primary studies' abstracts are representative of the full papers for the purpose of term frequency analysis.
“I created a bar chart of the most frequently used 50 terms in the primary studies' abstracts (Figure 5).”
The paper relies on this without support.
The classification of SE challenges based on the main SE knowledge areas presented in Wan et al. (2019) is suitable for organizing the identified challenges.
“The identified SE challenges are classified based on the main SE knowledge areas presented in (Wan et al., 2019).”
The paper relies on this without support.
The outputs of an ML model may be used by many other components implicitly.
“The outputs of an ML model may be used by many other components implicitly.”
The paper relies on this without support.
ML system performance is subject to change due to its non-deterministic nature.
“Moreover, ML systems should be monitored continuously since their performance is subject to change due to their non-deterministic nature.”
The paper relies on this without support.
ML models are dependent on ML frameworks.
“ML models are also dependent on ML frameworks.”
The paper relies on this without support.
Some large-scale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.
“While it is necessary to keep track of changes in data and ML models, some largescale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.”
The paper relies on this without support.
Each experiment may have many components that affect the outcome, i.e., the ML model.
“Each experiment may have many components that affect the outcome, i.e., the ML model.”
The paper relies on this without support.
It is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.
“Also, it is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.”
The paper relies on this without support.
One of the hardest parts of maintaining ML systems is to keep the performance (such as accuracy) at a certain level or improve it if needed.
“One of the hardest parts of maintaining ML systems is to keep the performance (such as accuracy) at a certain level or improve it if needed.”
The paper relies on this without support.
There are some process models, phases, activities that document steps to develop and maintain ML models.
“There are some process models, phases, activities that document steps to develop and maintain ML models.”
The paper relies on this without support.
Adding ML components to a software system involves additional actions to be performed.
“Adding ML components to a software system involves additional actions to be performed.”
The paper relies on this without support.
Having the required people with proper skill sets is necessary but not sufficient to engineer ML systems.
“Having the required people with proper skill sets is necessary but not sufficient to engineer ML systems.”
The paper relies on this without support.
Specifying ML components' requirements requires a different perspective than traditional software components.
“Specifying ML components' requirements requires a different perspective than traditional software components.”
The paper relies on this without support.
ML systems include traditional software components as well.
“Besides, ML systems include traditional software components as well.”
The paper relies on this without support.
Developers deal with additional complications while developing ML systems.
“Developers deal with additional complications while developing ML systems.”
The paper relies on this without support.
Another challenge is the estimation of resource requirements for production environment.
“Another challenge is the estimation of resource requirements for production environment.”
The paper relies on this without support.
Data, ML models, and experiments have become essential configuration items in maintaining ML systems.
“Data, ML models, and experiments have become essential configuration items in maintaining ML systems.”
The paper relies on this without support.
Keeping versions of data and ML models require tools integrated with the development environment.
“Keeping versions of data and ML models require tools integrated with the development environment.”
The paper relies on this without support.
Engineering ML systems is a highly practical area.
“although engineering ML systems is a highly practical area”
The paper relies on this without support.
ML systems engineering brings together two challenging research areas, i.e., SE and ML, and requires various skillsets.
“ML systems engineering brings together two challenging research areas, i.e., SE and ML, and requires various skillsets.”
The paper relies on this without support.
SE venues can provide a representative set to identify state of the art and challenges of engineering ML systems.
“However, I believe that SE venues can provide a representative set to identify state of the art and challenges of engineering ML systems.”
The paper relies on this without support.
More and more software systems are involving ML capabilities.
“More and more software systems are involving ML capabilities.”
The paper relies on this without support.
Advances in machine learning lead to a transition from conventional software development, where algorithms are hard-coded by humans, to ML systems that are materialized through learning from data.
“Advances in machine learning lead to a transition from the conventional view of software development, where algorithms are hard-coded by humans, to ML systems that are materialized through learning from data.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The quality of included studies was assessed using the criteria found in Table 2, adapted from criteria used by [18] and suggested by [17].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [18]
In the citing paper
“ML models have an additional data dependency, in the sense that their behavior is learned from data, and they have even been characterized as non-deterministic [18,36].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Executing tests:
“many safety violations identified through online testing”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Executing tests:
“could not be identified through online testing (Haq et al., 2020).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3. Analysis:
“This study 2007 -Dec 2020 identify, analyze, summarize, and synthesize the current state of SE research for engineering ML systems SMS using DB search, snowballing, and manual search 141 papers”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
“I performed a systematic literature review (SLR).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
“Due to these three primary studies using multiple research methods, the total number of primary studies equals 147 in Figure 6.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
“Two studies (Amershi et al., 2019;Wan et al., 2019) combined interview and survey methods.Two other studies (Humbatova et al., 2020;Zhang et al., 2020a) used thematic analysis and interview methods together.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
“Planning, controlling, and monitoring of testing activities for ML systems have not been addressed by researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Managing tests:
“In a case study, Rivero et al. (2020) utilized test management tools and spreadsheets to manage test process in which various teams were involved.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,191
(1994) · Cited by 21,711
Research works printed but not linked: 6
- Fujii, 2020, Guidelines for quality assurance of machine learning-based artificial intelligence, Int. J. Softw. Eng. Knowl. Eng., 1
- Wan, 2019, How does machine learning change software development practices?, IEEE Trans. Softw. Eng.
- Fei-Fei, L., 2010. ImageNet: crowdsourcing, benchmarking & other cool things. In: CMU VASC Seminar (Vol. 16). March. pp. 18-25.
- Schwerdtner, P., Grezner, F., Kapoor, N., Assion, F., Sass, R., Günther, W., Hüger, F., Schlicht, P., 2020. Risk assessment for machine learning models. In: Machine Learning for Autonomous Driving Workshop at the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R., 2014. Intriguing properties of neural networks. In: 2nd International Conference on Learning Representations (ICLR).
- Zaharia, 2018, Accelerating the machine learning lifecycle with mlflow, IEEE Data Eng. Bull., 41, 39
Cited by
Sorted by citations
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34
Metadata
- Authors
- Giray, G.
- Corresponding author
- Giray, Görkem
- ORCID
- Giray, Görkem
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2021-10
- Volume
- 180
- Pages
- 111031
- ISSN
- 0164-1212 (ISSN-L)
- OpenAlex
- W3112475255
- Cited by
- 187
- References
- 256
- Citations by year
- 2026: 23 · 2025: 37 · 2024: 38 · 2023: 43 · 2022: 36 · 2021: 10
- FWCI
- 28.71
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Journal of Systems and Software (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- software engineering for machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
