The DOI matches its Crossref record; 256 of the 256 references it cites were found in a registry or an index; none of the 326 sources its claims cite is checked for support yet.
A software engineering perspective on engineering machine learning systems: State of the art and challenges
Open accessOverview
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic literature review synthesizes 141 SE-venue primary studies (2007–Dec 2020) on engineering machine learning systems. It finds that ML systems’ non-deterministic nature complicates all SE aspects, no SE aspect yet has mature tools and techniques, and testing is the most studied area but still lacks strong experimental proof.
Key findings
- The review analyzed 141 primary studies from SE venues; research interest increased sharply from 2018 onward, and 91% of studies appeared in conferences/workshops (129 papers) versus 12 in journals.
- Testing is by far the most popular area among researchers, but even for testing ML systems engineers have only tool prototypes and solution proposals with weak experimental proof; many challenges were identified through surveys and interviews.
- Among 93 experiment/case-study papers, 65 (70%) used only DL algorithms, 19 used traditional ML, 4 used both, and 5 did not report algorithms; image classification was the most popular application scenario, used in 57 studies.
- No SE aspect has a mature set of tools and techniques, and the paper calls for experiments and case studies, ideally in industrial environments, to better understand challenges and propose solutions.
Methods
- Systematic literature review following established guidelines, with database search (ACM, IEEE Xplore, ScienceDirect, Springer, Wiley, Google Scholar), backward/forward snowballing, and manual search of ICSE, ESEC/FSE, AIRE, and RAISE.
- Inclusion/exclusion criteria restricted papers to SE venues and to those involving at least one SE challenge of engineering ML systems; after de-duplication and quality assessment, the final pool was 141 studies.
- Quantitative analysis reported frequencies/percentages for research methods and application scenarios/datasets; qualitative open coding analyzed challenges and solutions, classified under SWEBOK-based SE knowledge areas.
Limitations
- Paper-stated: scope is limited to studies published until 31 December 2020, peer-reviewed academic venues (grey literature excluded), and SE venues only (AI/ML/DL/Data Science/Data Management venues excluded); primary study selection, data extraction, and synthesis are subject to researcher bias, which may be higher in a single-author SLR.
- Paper-stated: single data extraction can produce more errors than double data extraction, and the author notes that selection only from SE venues may miss relevant studies from other venues.
- Evident from text: the search used broad terms and a “search saturation effect,” checking only the first 200 results per query in each database unless results 190–200 still looked relevant, which may limit recall despite snowballing and manual search.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
225 Not checked yet · 51 Own result · 23 No basis given · 20 Own method · 14 Own interpretation · 2 Sources not found
In the paper
“Unlike Kumeno (2019), I was able to identify more papers published in SE venues (141 vs. 47) and provided a taxonomy of challenges and proposed solutions classified under SE's main knowledge areas.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno (2019)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Testing is by far the most popular area among researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of the challenges of ML systems engineering were identified through surveys and interviews.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite increasing interest from 2018 onwards, the results reveal that none of the SE aspects have a mature set of tools and techniques.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This review on the state-of-the-art is the most contemporary as it is the most up-to-date review.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even for testing ML systems, engineers have only some tool prototypes and solution proposals with weak experimental proof.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The included number of papers in Kumeno (2019), Lwakatare et al. (2020), Nascimento et al. (2020), and this study are 115, 72, 55, and 141, respectively.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno (2019)
- Not checked yetcites Lwakatare et al. (2020) source
- Not checked yetcites Nascimento et al. (2020)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Eight studies in our primary study pool (Haq et al., 2020;Harel-Canada et al., 2020;Jahangirova and Tonella, 2020;Li et al., 2019b;Stocco and Tonella, 2020;Stocco et al., 2020;Tian et al., 2018;Zhang et al., 2018a) conducted an experiment for predicting steering wheel angle.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Haq et al., 2020
- Not checked yetcites Harel-Canada et al., 2020
- Not checked yetcites Jahangirova and Tonella, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Image classification is the most popular application scenario used in 57 primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Twenty studies included a classification scenario other than image classification.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 62
- Probably not a substantial claim. 155
- The quote only partly states this claim. 1
In the paper
“While none of the existing reviews addressing SE aspects (Kumeno, 2019;Lwakatare et al., 2020;Nascimento et al., 2020) cover the papers published in 2020, this study covers the papers published until the end of 2020 in SE venues (75 papers in total).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3 RELATED WORK
What the sources say
- Not checked yetcites Kumeno, 2019;Lwakatare et al., 2020;Nascimento et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to these three primary studies using multiple research methods, the total number of primary studies equals 147 in Figure 6.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thematic analysis, case study, and statistical analysis are the other approaches used by the researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Four studies mention ML systems predicting values for combined sewer overflow (Challa et al., 2020), stock price (Wang et al., 2020b), age (Gao et al., 2020a), real estate value and weather (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Challa et al., 2020
- Not checked yetcites Wang et al., 2020b
- Not checked yetcites Gao et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One case study (Nakamichi et al., 2020) addressing requirements engineering challenges did not report its application scenario(s).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Nakamichi et al., 2020
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 8 shows the distribution of application scenarios used in 93 of the primary studies, the research method of which is either an experiment or a case study.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“One of their experiments involved BERT (Devlin et al., 2018) model over GLUE (General Language Understanding Evaluation) benchmark (Wang et al., 2018), with various batch sizes and sequence lengths.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Devlin et al., 2018
- Not checked yetcites Wang et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Liu et al. (2020b) explored technical debts in seven open-source DL frameworks and their impacts on the maintainability of DL systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Liu et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Three studies involved an ML system for object detection.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides experiments, the researchers interviewed and surveyed experts to learn about the challenges they face in engineering ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Tizpaz-Niari et al. (2020) discovered performance bugs in ML frameworks, such as the implementation of logistic regression in scikit-learn.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Tizpaz-Niari et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Only one study (Murphy et al., 2007) is concerned with a ranking problem.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Two other studies (Humbatova et al., 2020;Zhang et al., 2020a) used thematic analysis and interview methods together.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Humbatova et al., 2020
- Not checked yetcites Zhang et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Sun and Zhou (2018) and Zheng et al. (2019) addressed the challenges of testing an ML system on machine translation.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Sun and Zhou, 2018
- Not checked yetcites Zheng et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There are also many terms representing the intervention of this study, i.e.,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“I divided the included studies into empirical (127 papers) and non-empirical (15 papers), as suggested by Mohanani et al. (2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
What the sources say
- Not checked yetcites Mohanani et al. (2018)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Ahmed et al. (2020), Dutta et al. (2020), and Wang et al. (2020b) studied problems in DL/ML frameworks related to testing and quality.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Ahmed et al., 2020
- Not checked yetcites Dutta et al., 2020
- Not checked yetcites Wang et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Two studies (Amershi et al., 2019;Wan et al., 2019) combined interview and survey methods.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can observe many terms that represent the population of this study, i.e.,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Alshangiti et al. (2019) performed thematic and statistical analysis together.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Murphy et al. (2007) aim at ranking the probabilities of observing a failure in a device.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Pham et al. (2020) conducted experiments to study the variance of DL systems and a survey to understand the awareness of this variance among researchers and practitioners.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
What the sources say
- Not checked yetcites Pham et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three studies contained application scenarios of Natural Language Processing, two specifically machine translation.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fifteen papers either provide the author's opinions or did not report enough details about the research method.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Trujillo et al. (2020) used mountain car problem to investigate whether neuron coverage can be used for deep reinforcement learning systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Trujillo et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 7 shows the list of datasets used more than once in the experiments and case studies in 93 primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Gao et al. (2020b) proposed an estimation tool for GPU memory consumption of DL models to help developers in planning runtime resource requirements.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Gao et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally yet importantly,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not surprisingly,”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They reported one of the complexities posed by ML systems as the”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Sculley et al., 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Washizaki et al. (2019b) state that ad-hoc solutions are being used for solving everyday problems in developing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Washizaki et al., 2019b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Wan et al. (2019) interviewed professionals with experience in both traditional and ML system development and reported that nearly every interviewee made a strong statement about differences between the requirements of ML versus non-ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.1 Requirements Engineering
What the sources say
- Not checked yetcites Wan et al., 2019
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Benton et al. (2020) proposes an architecture for an ML system that deals with high-volume data.This architecture includes a component for data federation to deal with structured, unstructured, and streaming data.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Benton et al., 2020 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For instance, Jiarpakdee et al. (2020) argued that the lack of explainability of defect prediction models could hinder the adoption of these models in practice.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Jiarpakdee et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“184 datasets were reported in 93 studies, which means most of the studies used two datasets on average.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“John et al. (2020a; 2020b) proposed a framework involving five architectural alternatives ranging from centralized cloud to fully decentralized edge architectures for designing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites John et al., 2020a
- Not checked yetcites John et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Yokoyama (2019) proposes an architectural pattern for locating performance problems and rolling back in case of a failure on production.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Yokoyama, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Washizaki et al. (2020) identified 15 design patterns for ML systems and plan to expand the list via further empirical studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Washizaki et al., 2020
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“To provide more comprehensive knowledge at different level of expertise, Hashemi et al. (2020) automatically generated ML system documentation from StackOverflow Q&As.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Hashemi et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“I classified these challenges into eight categories.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, the top SE conferences, i.e., ESEC/FSE and ICSE, receive submissions on this emerging topic.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The number of primary studies in SE venues on engineering ML systems has approximately doubled annually every year since 2018.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One hundred and one papers (72%) were authored solely by academic researchers, 11 papers (8%) by researchers from research organizations or companies, and 29 papers (20%) were written jointly by universities and other organizations.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Development & tools and requirements engineering areas the second and third areas in which researchers collaborated.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three papers were published in 2017, after which we see a sharp increase starting from 2018.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, researchers collaborated the most within the scope of testing and quality.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The earliest paper (Murphy et al., 2007) was published at the International Conference on Software Engineering & Knowledge Engineering in 2007.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.4 DEMOGRAPHICS
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Regression testing of ML systems has not been addressed in the primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“SE researchers focused mostly on testing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Planning, controlling, and monitoring of testing activities for ML systems have not been addressed by researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The conference with the most papers (20) is the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Researchers have been showing an increasing interest in this research area since 2018 (Figure 9).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 7 CONCLUSIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Case studies enabled researchers to collect data on activities related to ML systems in the industry.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, experts mentioned many challenges covering all aspects of engineering ML systems in interviews and surveys.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure4, the DB search yielded 90 results, snowballing 68, and manual search 58.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plainn = 141- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The journals with the most papers (three in each) are IEEE Access, IEEE Software, and IEEE Transactions on Software Engineering.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“91% of the studies were presented in conferences and workshops (129 papers in conferences/workshops and 12 papers in journals).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nineteen papers used traditional ML algorithms, and four papers used both DL and traditional ML algorithms.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although there are a lot of primary studies (Section 9.1) and secondary studies (Section 3) on this topic, many research questions remain unanswered (Section 6.1 and Table 8).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 7 CONCLUSIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were 77 studies found via only one search method (DB search: 40, Snowballing: 31, Manual search: 6) and 64 studies via more than search method.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Number of Primary Studies Reference(s)
plainn = 141- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Many researchers from software engineering (SE) (Hesenius et al., 2019;Northrop et al., 2019) and ML (Karpathy, 2017;Menzies, 2019), as well as industry practitioners (Benton, 2020;Booch, 2019;Lin and Kolcz, 2012) have stated the requirement of such a holistic view.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 1 INTRODUCTION
What the sources say
- Not matchedcites Hesenius et al., 2019;Northrop et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Karpathy, 2017;Menzies, 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Benton, 2020;Booch, 2019;Lin and Kolcz, 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“The research method was based on established guidelines (Kitchenham and Charters, 2007;Wohlin, 2014), some previous good examples (Durelli et al., 2019;Mohanani et al., 2018), and my previous experience in conducting SLR (Garousi et al., 2019;Giray and Tüzün, 2018;Tarhan and Giray, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4 RESEARCH OBJECTIVES AND METHOD
What the sources say
- Not matchedcites Kitchenham and Charters, 2007;Wohlin, 2014
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Durelli et al., 2019;Mohanani et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Garousi et al., 2019;Giray and Tüzün, 2018;Tarhan and Giray, 2017
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To maintain a high-quality input of primary studies for this SLR, I decided to exclude the papers with a score lower than six points out of 12 for empirical studies and three points out of six for non-empirical studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“To assess the quality of primary studies, I used the quality assessment criteria (Table 5) proposed by Mohanani et al. (2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
What the sources say
- Not checked yetcites Mohanani et al. (2018)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all primary studies, I recorded the challenges and solutions to the spreadsheet without making any changes as written in the primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“I used the knowledge areas presented in Wan et al. (2019), which was derived from SWEBOK (Bourque and Fairley, 2014) to classify the challenges and solutions.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
What the sources say
- Not checked yetcites Wan et al. (2019)
- Not checked yetcites Bourque and Fairley, 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“As briefly explained in Section 2.2, traditional ML algorithms entail a manual feature selection process, which may require domain expertise, statistical analysis of input data, and experiments for building models with different feature sets (Kelleher, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.2 WHAT APPLICATION SCENARIOS AND DATASETS WERE USED FOR IN EXPERIMENTS AND CASE STUDIES?
What the sources say
- Not checked yetcites Kelleher, 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“As these examples show, the industry is calling for action to resolve the challenges of engineering ML systems and to propose new techniques to cope with the additional complexity of ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > SE
What the sources say
- Not checked yetcites Lorica and Loukides (2018)
- Not checked yetcites Heck, 2019
- Not checked yetcites Sato et al. (2019)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“showing researchers' high interest in testing ML systems”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5 RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“I conducted open coding (Miles and Huberman, 1994) to analyze the challenges and solutions.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.4 DATA SYNTHESIS AND REPORTING
What the sources say
- Not checked yetcites Miles and Huberman, 1994
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I excluded one study with a score under the threshold.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 4.2.6 Quality assessment
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This change poses two crucial challenges: detecting when concept drift occurs and keeping the models up-to-date (Maimon and Rokach, 2010).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Maimon and Rokach, 2010
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“There is a need for a catalog of patterns dedicated to ML-specific problems (Washizaki et al., 2019b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Washizaki et al., 2019b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Requirements of ML systems are heavily dependent on available data; different data sets may lead to different needs (Rahimi et al., 2019;Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Eliciting and analyzing requirements:
What the sources say
- Not checked yetcites Rahimi et al., 2019
- Not checked yetcites Wan et al., 2019
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Requirements are much more uncertain (Belani et al., 2019;Chechik, 2019;Ishikawa and Yoshioka, 2019;Rahimi et al., 2019;Wan et al., 2019), and hence they are more difficult to elicit and analyze.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Eliciting and analyzing requirements:
What the sources say
- Not checked yetcites Belani et al., 2019
- Not checked yetcites Chechik, 2019
- Not checked yetcites Ishikawa and Yoshioka, 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Explainability is quite essential in building trust between an ML system and its users (Nguyen-Duc et al., 2020;Ribeiro et al., 2016).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Specifying requirements:
What the sources say
- Not checked yetcites Nguyen-Duc et al., 2020
- Not checked yetcites Ribeiro et al., 2016
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition, combining data is not always straightforward (Kim et al., 2017;Lwakatare et al., 2019) since different systems can have other objectives for storing data, hence leading to data semantics heterogeneity and data integration problems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites Kim et al., 2017 source
- Not checked yetcites Lwakatare et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Distributed architectural patterns are widely used in designing ML systems to cope with high-volume data (Wan et al., 2019), which leads to additional complexity in architectural and detailed design (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Wan et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Known as”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites Tsymbal, 2004
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Trade-offs among device cost, ML model performance, and data privacy requirements guide the selection of an architectural alternative (John et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Designing for monitoring performance degradation on production:
What the sources say
- Not checked yetcites John et al., 2020b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They may need to optimize a possible tradeoff between the best features and model complexity (Sapp, 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Sapp, 2017
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although non-testability is attributed to a lack of reliable test oracle (Weyuker, 1982), there are several challenges in testing ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Weyuker, 1982 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Model deployment involves some challenges (Chen et al., 2020b).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Chen et al., 2020b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Adapting the implementation of a neural network for a different task (Zhang et al., 2019a) or transferring a built solution to another domain (Baier et al., 2019) remains a challenge.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Zhang et al., 2019a
- Not checked yetcites Baier et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Another essential concern in developing ML models is fairness (Corbett-Davies and Goel, 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Corbett-Davies and Goel, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even the model requirements may not be apparent before starting with model development and may be clarified after many experiment iterations.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“They face challenges in understanding these (Alshangiti et al., 2019;Zhang et al., 2019a;Zhang et al., 2020a), especially the ones related to DL (Alshangiti et al., 2019), and identify a proper model (Islam, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Alshangiti et al., 2019
- Not checked yetcites Zhang et al., 2019a
- Not checked yetcites Zhang et al., 2020a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This development endeavor involves some sub-activities, such as feature engineering, model training, model evaluation, and model deployment (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In 2007, Murphy et al. (2007) mentioned the idea of testing ML systems and classified such systems as”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Murphy et al., 2007
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Zhou et al. (2020b) found out that OoD-aware ML models are more robust according to their experiments.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Zhou et al., 2020b
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While reusing ML libraries is very common, effective, and efficient, reusing ML models in different domains or systems is not straightforward (Alvarez-Rodríguez et al., 2019;Amershi et al., 2019;Baier et al., 2019;Guo et al., 2019;Sankaran et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Alvarez-Rodríguez et al., 2019
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Baier et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For DL models, feature engineering is performed implicitly during model training (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Amershi et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Distributed systems require further knowledge and experience and additional cost and management of associated hardware and software (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the development environment, tools, and infrastructure:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For instance, imbalanced datasets will most likely lead to inaccurate prediction models (He and Garcia, 2009).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.3 Software Development and Tools
What the sources say
- Not checked yetcites He and Garcia, 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Welldesigned test cases can help in finding many types of defects in ML systems (Humbatova et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Humbatova et al., 2020
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Nevertheless, DL models should be evaluated using various important metrics such as robustness and neuron coverage, besides the widely-used prediction accuracy metric (Tian et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Understanding ML algorithms, techniques, and libraries:
What the sources say
- Not checked yetcites Tian et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Galhotra et al. (2017) proposed a technique to measure causal discrimination and managed to detect significant ML model discrimination towards gender, marital status, or race as many as 77.2% of the individuals in datasets.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Galhotra et al., 2017
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For instance, according to the experiments conducted by Haq et al. (2020), simulator-generated datasets yielded DNN prediction errors that are similar to those obtained via real-world datasets in autonomous driving domain.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Haq et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The discovery of adversarial examples and evaluation of model robustness are fundamental problems in engineering ML systems (Du et al., 2020;Fujii et al., 2020;Sun et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Du et al., 2020
- Not checked yetcites Fujii et al., 2020
- Not checked yetcites Sun et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Zhou et al. (2020a) presented their approach named DeepReduce, which helped them reduce testing data 7.5% on average and still reliably estimate the performance of DL models.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Zhou et al., 2020a
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Based on their preliminary results, neuron coverage is not sufficient to reach reliable conclusions about the quality DeepRL models (Trujillo et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Trujillo et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Yan et al. (2020) claimed that DNN coverage criteria do not have monotonic relations with model quality measured by model accuracy in the presence of adversarial examples.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Yan et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They classified these bugs into 13 root causes, 20 bug symptoms, and 18 categories of software components these bugs often affect (Garcia et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Garcia et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Mutation Testing (MuT) is a fault-based testing technique that claims to address the shortcomings of coverage criteria (Jia and Harman, 2010;Zhang et al., 2018c).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Jia and Harman, 2010
- Not checked yetcites Zhang et al., 2018c
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Based on the experiments conducted by Harel-Canada et al. (2020), higher neuron coverage led to fewer defects detected, less natural inputs, and more biased prediction preferences.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Harel-Canada et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Kim et al. (2019) and Sekhon and Fleming (2019) state that existing coverage criteria are not sufficiently fine-grained to capture subtle behaviors exhibited by ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Kim et al., 2019
- Not checked yetcites Sekhon and Fleming, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although researchers try to develop new coverage criteria appropriate for DL models, Li et al. (2019) could not find a strong correlation between the number of misclassified inputs in a test set and its structural coverage.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Li et al., 2019
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The size of test case sets is enormous, and missing test cases is a more challenging problem for ML systems (Khomh et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Khomh et al., 2018 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Flaky tests are observed frequently and may cause problems in software development process (Luo et al., 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Luo et al., 2014
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.4 Testing and Quality
What the sources say
- Not checked yetcites Zhou et al., 2015 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Failing to update the code logic accordingly as ML models evolve may introduce bugs and fail to achieve the optimal performance of ML models, and further negatively influence the entire ML-powered system (Peng et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Peng et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such mechanisms can detect performance degradation and take the required actions by re-training ML models with new data.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For instance, Hulten (2018) breaks down the ML process into five stages: (1) getting data to model, (2) feature engineering, (3) modeling, (4) deployment, and (5) maintenance.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Hulten, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Shams (2018) demonstrates a threephase ML product development process to shorten the delivery time from idea generation to deployment.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Shams, 2018 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Canary release approach may limit the possible negative impact of the new ML model and help to cope with the risk of deploying an updated ML model (Yokoyama, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Yokoyama, 2019
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“These components can include hardware (GPU models primarily), platform (operating system and installed packages), source code (model training and pre-processing), configuration (model configuration and pre-processing settings), training data (input signals and target values), and model state (versions of trained models) (Arpteg et al., 2018;Wan et al., 2019;Yokoyama, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Arpteg et al., 2018
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Yokoyama, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Each experiment may have many components that affect the outcome, i.e., the ML model.
In the paper
“They found that developers are mostly fixing data dimensions and neural network connectivity (Islam et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Islam et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Rao (2019) shows the typical steps involved in the model development lifecycle: (1) define the problem, (2) collect, cleanse, and prepare data, (3) build and train model, (4) tune hyperparameters and validate model, and (5) deploy to production.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Rao, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Therefore, experiment management is a recent and complicated challenge in engineering ML systems (Hazelwood et al., 2018;Zaharia et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Hazelwood et al., 2018
- Not checked yetcites Zaharia et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Some large-scale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.
- Rests on an assumption: Each experiment may have many components that affect the outcome, i.e., the ML model.
- Rests on an assumption: It is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.
In the paper
“While versioning data is a vital need (Alvarez-Rodríguez et al., 2019;Amershi et al., 2019;Belani et al., 2019;Wan et al., 2019;Yokoyama, 2019), the methods and tools to support this is not mature yet (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Peng et al. (2020) observed that as ML models evolve, it poses maintenance challenges on the code logic that should coevolve.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Peng et al., 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For instance, Kubernetes ecosystem may ease maintenance and problem localization (Benton, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Benton, 2020 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“While developers fix bugs, they may introduce adversarial vulnerabilities and new bugs (Islam et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Islam et al., 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) experiments and their results may not be analyzed, reported, and shared appropriately.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The quote only partly states this claim.
In the paper
“Whether done automatically or manually, experimentation activities generate many artifacts (datasets, hyperparameters, models, libraries, etc.), which should be versioned (Golovin et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Golovin et al., 2017
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“In risky domains, automated checks can be used to trigger notifications if predefined thresholds are violated (Baier et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Baier et al., 2019
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Versioning data does not only involve keeping track of datasets but also keeping track of metadata of datasets (Gebru et al., 2018) and their relationships to ML models (Amershi et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Gebru et al., 2018
- Not checked yetcites Amershi et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The steps are not executed as a waterfall process; there is feedback from deployment to problem definition and an iteration from building a model to deployment (Rao, 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Rao, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“In another study, Pan (2020) observed that bug fixes may cause both an increase and decrease in robustness of DL models.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Debugging and fixing:
What the sources say
- Not checked yetcites Pan, 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Another option is to augment training data with the guidance of newly identified input data and retrain ML model (Ren et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Ren et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“With the emergence of many general-purpose ML libraries, services, platforms combined with the accumulation of ML experience, developing ML systems has become relatively easy and cheap, whereas maintaining them over time is difficult and expensive (Sculley et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.5 Maintenance and Configuration Management
What the sources say
- Not checked yetcites Sculley et al., 2015
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Newly identified data can be used directly for adaptive retraining (Stocco and Tonella, 2020).”
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
“The identified data can be used for adaptive retraining.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Dealing with the history of experiments:
What the sources say
- Not checked yetcites Stocco and Tonella, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE · plain
In the paper
“An ML engineer need to understand the mathematical and statistical background of models for proper model fitting and tuning (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Moreb et al. (2020) proposed a process for engineering ML systems in health informatics domain.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Moreb et al., 2020 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A software engineer may have to deal with distributed systems, web services, and multi-threaded programming (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Due to the uncertainties in the ML process, it is hard to estimate effort and do planning (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Estimating effort:
What the sources say
- Not checked yetcites Wan et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The popularity of ML has brought many tools, mostly open-source (Louridas and Ebert, 2016).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Louridas and Ebert, 2016
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“According to Sensemaking-Coevolution-Implementation Theory, sensemaking is one of the core activities in engineering software systems (Ralph, 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Ralph, 2015 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For example, while data scientists may be pragmatic when coding as long as they achieve the desired results, software engineers may care about code quality and maintainability (Arpteg et al., 2018).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Arpteg et al., 2018
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Components with ML capabilities are becoming an architectural part of software systems, sharing cross-cutting functional and non-functional concerns (Nguyen-Duc et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Nguyen-Duc et al., 2020
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“People having different backgrounds have cultural (Arpteg et al., 2018) and language differences.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Building harmony among isolated roles:
What the sources say
- Not checked yetcites Arpteg et al., 2018
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Engineering complex systems, such as ML systems, is a complicated endeavor with organizational and technical aspects (Clarke et al., 2016).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Clarke et al., 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Wolf and Paine (2020) emphasizes the difficulty of sensemaking for business context, ML algorithms, libraries, and models in ML systems engineering projects.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Wolf and Paine, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Alvarez-Rodríguez et al. (2019) identified the formation of a workforce equipped with the required skills to tackle different types of problems in various domains as a challenge.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alvarez-Rodríguez et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A data preparer requires a good data slicing and dicing skills for proper data cleaning and feature preprocessing (Alshangiti et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Alshangiti et al., 2019
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Monitoring for potential performance degradation on production (Wan et al., 2019;Yokoyama, 2019) and high-volume data processing (de Souza Nascimento et al., 2019;Wan et al., 2019) are two important design considerations for ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Wan et al., 2019
- Not checked yetcites Yokoyama, 2019
- Not checked yetcites de Souza Nascimento et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reason for having much fewer papers in journals may be due to the fact that this topic is still in its infancy.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6 DISCUSSION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“There are some early attempts to determine the ML process' maturity (Amershi et al., 2019) and how ML components are used in larger software systems (Lwakatare et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.6 Software Engineering Process and Management
What the sources say
- Not checked yetcites Amershi et al., 2019
- Not checked yetcites Lwakatare et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“There are some efforts to bring various tools together to support the effective development of ML systems (Smith et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Smith et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“However, this diverse set of tools have emerged with challenges, such as compatibility (Rahman et al., 2019), integration of various components (Rahman et al., 2019), required knowledge and experience, etc.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rahman et al., 2019
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“ML systems' development involves a diverse set of frameworks, tools, libraries, and programming languages (Rahman et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rahman et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“While there are generic roles such as data scientist, ML engineer, data analyst, ML software engineer, ML researcher, and software engineer, there are more specific roles, such as data evangelist, data preparer, data shaper, data analyzer (Kim et al., 2017).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.3.7 Organizational Aspects
What the sources say
- Not checked yetcites Kim et al., 2017 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I used broad terms to form search keywords to reduce the risk of excluding potentially relevant studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Date: This study covers primary studies published until the end of 2020, i.e., 31 December 2020.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“However, the success of these coverage criteria is questionable (Li et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Li et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“To maintain the performance of an ML system above a level that meets customer expectations, drift detectors may be developed to identify data deviating from training data distribution (Stocco and Tonella, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Stocco and Tonella, 2020
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A set of guidelines was published to ensure ML systems' quality (QA4AI, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites QA4AI, 2020
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Perspective: The primary studies have been selected from SE venues to reflect an SE perspective on ML systems engineering.Therefore, papers published in AI, ML, DL, Data Science, Data Management venues have been excluded.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“With this approach, I decreased precision and increased recall found more candidate papers to be assessed for specifying the final set of primary studies.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“It discusses how to take an idea and an ML model developed by data scientists and deploy it as part of a scalable and maintainable system.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites Kästner and Kang, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Rudraraju and Boyanapally, (2019) focused on developing a data quality model for ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rudraraju and Boyanapally, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Personal bias might have been introduced during the application of inclusion and exclusion criteria.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They identified new data quality attributes that are not present in the Data Quality model defined in the standard ISO/IEC 25012.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Rudraraju and Boyanapally, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“As an example of the fulfillment of this need, the QA4AI Consortium was established in 2018 in Japan to discuss the quality assurance of ML systems (Fujii et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Fujii et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of the concerns, such as some non-functional requirements, may be domain-specific (Nguyen-Duc et al., 2020) and can be discovered by exploring various ML problems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Use of various application scenarios:
What the sources say
- Not checked yetcites Nguyen-Duc et al., 2020
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The 2020 version of the Future of Jobs Survey reveals that the top increasingly strategic job roles are data analysts and scientists, AI and ML specialists, robotics engineers, software and application developers, and digital transformation specialists (World Economic Forum, 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites World Economic Forum, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Type of Literature: This study comprises studies published in peer-reviewed academic venues.Grey literature, e.g., papers only published in arxiv.org,blogs, videos, etc., have been excluded.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate the risk of missing some relevant studies, I also carried out both backward and forward snowballing along with a manual search.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Tools to estimate computation and memory requirements, such as DNNMem (Gao et al., 2020b), may help developers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Gao et al., 2020b
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I also searched for five widely used online databases in SMS and SLR studies in computer science and software engineering.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Researchers analyzed and classified these bugs (Garcia et al., 2020).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Garcia et al., 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One of the pioneer courses is the”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites Kästner and Kang, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Automating tests has become a necessity to cope with large-scale complex systems (Mosley and Posey, 2002).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Mosley and Posey, 2002
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While there are many processes for ML lifecycle (Hulten, 2018;Rao, 2019;Shams, 2018) and traditional software development (Klünder et al., 2019), a set of harmonized practices for developing ML systems is required (Alvarez-Rodríguez et al., 2019;Belani et al., 2019;de Souza Nascimento et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Hulten, 2018
- Not checked yetcites Rao, 2019
- Not checked yetcites Shams, 2018 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Wan et al.only mentioned the difficulty of effort estimation in ML projects (Wan et al., 2019).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Wan et al., 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While there are some examples of using ML algorithms for project management (two examples in ML for SE by Nassif et al. (2016) and Pospieszny et al. (2018)), our primary study pool does not include any practices for managing ML systems development projects.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Validity considerations are applicable for SMS and SLR studies, like empirical studies (Petersen et al., 2008;Petersen et al., 2015).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
What the sources say
- Not checked yetcites Petersen et al., 2008
- Not checked yetcites Petersen et al., 2015 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There may be other relevant studies published in AI, ML, DL, Data Science, and Data Management venues.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One of the challenges to be addressed is the identification of metamorphic relations that should be satisfied by an ML system (Kanewala and Bieman, 2014).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Kanewala and Bieman, 2014 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The course takes a SE perspective on building software systems with a significant AI/ML component.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > For educators:
What the sources say
- Not checked yetcites Kästner and Kang, 2020
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides these five databases, I also searched Google Scholar to enrich the paper pool.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More testing can be conducted via test automation to reduce the risks of ML systems.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Two promising techniques for designing test cases are metamorphic testing techniques and leveraging adversarial techniques to detect erroneous behavior (Mjeda and Botterweck, 2019) on which further research is needed.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
What the sources say
- Not checked yetcites Mjeda and Botterweck, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The threats to this SLR's validity are mainly related to the specification of the candidate pool of papers, primary study selection bias, data extraction, and data synthesis.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To minimize this type of bias and errors whenever I could not decide on inclusion/exclusion by reading the abstract, I scanned the full text for the final decision.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The validity of data extraction is another essential aspect that directly affects this study's results.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I applied the open coding technique iteratively and incrementally to identify the challenges and solutions (RQ3).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This coding process potentially entails some researcher bias.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, primary study selection, data extraction, and synthesis are subject to researcher bias, and this bias may be higher than those SLRs with multiple authors.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to ensure the accuracy of the extracted data, I used existing categories in the literature where possible (RQ1, RQ2, and SE knowledge areas in RQ3).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“I aimed to decrease the risk of researcher bias by mapping the relevant data in primary studies to the specified groups.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Despite its disadvantages, we can find SLRs with single authors in SE (Malhotra, 2015) and other research areas (Mitchell, 2011).”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.4 LIMITATIONS AND POTENTIAL THREATS TO VALIDITY
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
The search space can be restricted by assuming a search saturation effect, such that checking the first 200 results of each query in each database is sufficient.
“Therefore, as others have done (Adams et al., 2016;Garousi and Mäntylä, 2016;Garousi et al., 2018;Godin et al., 2015), I had to restrict the search space by assuming a”
The paper relies on this without support.
The primary studies' abstracts are representative of the full papers for the purpose of term frequency analysis.
“I created a bar chart of the most frequently used 50 terms in the primary studies' abstracts (Figure 5).”
The paper relies on this without support.
The classification of SE challenges based on the main SE knowledge areas presented in Wan et al. (2019) is suitable for organizing the identified challenges.
“The identified SE challenges are classified based on the main SE knowledge areas presented in (Wan et al., 2019).”
The paper relies on this without support.
The outputs of an ML model may be used by many other components implicitly.
“The outputs of an ML model may be used by many other components implicitly.”
The paper relies on this without support.
ML system performance is subject to change due to its non-deterministic nature.
“Moreover, ML systems should be monitored continuously since their performance is subject to change due to their non-deterministic nature.”
The paper relies on this without support.
ML models are dependent on ML frameworks.
“ML models are also dependent on ML frameworks.”
The paper relies on this without support.
Some large-scale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.
“While it is necessary to keep track of changes in data and ML models, some largescale ML systems involve configuration management at a higher level, i.e., keeping track of experiments.”
The paper relies on this without support.
Each experiment may have many components that affect the outcome, i.e., the ML model.
“Each experiment may have many components that affect the outcome, i.e., the ML model.”
The paper relies on this without support.
It is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.
“Also, it is possible to use automated metaoptimization methods and conduct lots of experiments without human intervention.”
The paper relies on this without support.
One of the hardest parts of maintaining ML systems is to keep the performance (such as accuracy) at a certain level or improve it if needed.
“One of the hardest parts of maintaining ML systems is to keep the performance (such as accuracy) at a certain level or improve it if needed.”
The paper relies on this without support.
There are some process models, phases, activities that document steps to develop and maintain ML models.
“There are some process models, phases, activities that document steps to develop and maintain ML models.”
The paper relies on this without support.
Adding ML components to a software system involves additional actions to be performed.
“Adding ML components to a software system involves additional actions to be performed.”
The paper relies on this without support.
Having the required people with proper skill sets is necessary but not sufficient to engineer ML systems.
“Having the required people with proper skill sets is necessary but not sufficient to engineer ML systems.”
The paper relies on this without support.
Specifying ML components' requirements requires a different perspective than traditional software components.
“Specifying ML components' requirements requires a different perspective than traditional software components.”
The paper relies on this without support.
ML systems include traditional software components as well.
“Besides, ML systems include traditional software components as well.”
The paper relies on this without support.
Developers deal with additional complications while developing ML systems.
“Developers deal with additional complications while developing ML systems.”
The paper relies on this without support.
Another challenge is the estimation of resource requirements for production environment.
“Another challenge is the estimation of resource requirements for production environment.”
The paper relies on this without support.
Data, ML models, and experiments have become essential configuration items in maintaining ML systems.
“Data, ML models, and experiments have become essential configuration items in maintaining ML systems.”
The paper relies on this without support.
Keeping versions of data and ML models require tools integrated with the development environment.
“Keeping versions of data and ML models require tools integrated with the development environment.”
The paper relies on this without support.
Engineering ML systems is a highly practical area.
“although engineering ML systems is a highly practical area”
The paper relies on this without support.
ML systems engineering brings together two challenging research areas, i.e., SE and ML, and requires various skillsets.
“ML systems engineering brings together two challenging research areas, i.e., SE and ML, and requires various skillsets.”
The paper relies on this without support.
SE venues can provide a representative set to identify state of the art and challenges of engineering ML systems.
“However, I believe that SE venues can provide a representative set to identify state of the art and challenges of engineering ML systems.”
The paper relies on this without support.
More and more software systems are involving ML capabilities.
“More and more software systems are involving ML capabilities.”
The paper relies on this without support.
Advances in machine learning lead to a transition from conventional software development, where algorithms are hard-coded by humans, to ML systems that are materialized through learning from data.
“Advances in machine learning lead to a transition from the conventional view of software development, where algorithms are hard-coded by humans, to ML systems that are materialized through learning from data.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The quality of included studies was assessed using the criteria found in Table 2, adapted from criteria used by [18] and suggested by [17].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [18]
In the citing paper
“ML models have an additional data dependency, in the sense that their behavior is learned from data, and they have even been characterized as non-deterministic [18,36].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Executing tests:
“many safety violations identified through online testing”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Executing tests:
“could not be identified through online testing (Haq et al., 2020).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 3. Analysis:
“This study 2007 -Dec 2020 identify, analyze, summarize, and synthesize the current state of SE research for engineering ML systems SMS using DB search, snowballing, and manual search 141 papers”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Abstract
“I performed a systematic literature review (SLR).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
“Due to these three primary studies using multiple research methods, the total number of primary studies equals 147 in Figure 6.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 5.1 WHAT RESEARCH METHODS WERE USED?
“Two studies (Amershi et al., 2019;Wan et al., 2019) combined interview and survey methods.Two other studies (Humbatova et al., 2020;Zhang et al., 2020a) used thematic analysis and interview methods together.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > 6.1 POTENTIAL RESEARCH DIRECTIONS FROM THE SE PERSPECTIVE
“Planning, controlling, and monitoring of testing activities for ML systems have not been addressed by researchers.”
§ A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges > Managing tests:
“In a case study, Rivero et al. (2020) utilized test management tools and spreadsheets to manage test process in which various teams were involved.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,191
(1994) · Cited by 21,711
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
(2016) · Cited by 11,679
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00907 · Cited by 6,875
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234
Systematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239
Recent Trends in Deep Learning Based Natural Language ProcessingPreprint
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · IEEE Computational Intelligence Magazine · doi:10.48550/arxiv.1708.02709 · Cited by 3,133
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Data Mining and Knowledge Discovery Handbook
(2010) · doi:10.1007/978-0-387-09823-4 · Cited by 2,190
NewsWeeder: Learning to Filter Netnews
Lang, K. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50048-7 · Cited by 2,119
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876
Patterns of Enterprise Application Architecture
Fowler, M. (2002) · Cited by 1,809
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,752
Requirements Engineering: Processes and Techniques
Kotonya, G., & Sommerville, I. (1998) · Lancaster EPrints (Lancaster University) · Cited by 1,702
(2018) · International Conference on Learning Representations · Cited by 1,621
Agile Software Development, Principles, Patterns, and Practices
Martin, R. (2002) · Cited by 1,466
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180220 · Cited by 1,460
Scaling Up the Accuracy of Naive-Bayes Classifiers : a Decision-Tree Hybrid
Kohavi, R. (1996) · Cited by 1,418
Machine learning in manufacturing: Advantages, challenges, and applications
Wuest, T., Weimer, D., Irgens, C., & Thoben, K.-D. (2016) · Production & Manufacturing Research · doi:10.1080/21693277.2016.1192517 · Cited by 1,319
Design and Use of Software Architectures: Adopting and Evolving a Product-Line Approach
Bosch, J. (2000) · Medical Entomology and Zoology · Cited by 1,167
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Fastai: A Layered API for Deep Learning
Howard, J., & Gugger, S. (2020) · Information · doi:10.3390/info11020108 · Cited by 1,055
Deep Learning for Finance: Deep Portfolios
Heaton, J. B., Polson, N., & Witte, J. H. (2016) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.2209 · Cited by 1,047
A data-driven approach to predict the success of bank telemarketing
Moro, S., Cortez, P. C., & Rita, P. (2014) · Decision Support Systems · doi:10.1016/j.dss.2014.03.001 · Cited by 976
Automated software test data generation
Korel, B. (1990) · IEEE Transactions on Software Engineering · doi:10.1109/32.57624 · Cited by 973
Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0125-0 · Cited by 959
The problem of concept drift: definitions and related work
Tsymbal, A. (2004) · Cited by 929
(2015) · neural information processing systems · Cited by 912
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
McMahan, H. B., Holt, G., Sculley, D., Young, M., Ebner, D., Grady, J., Nie, L., Phillips, T., Davydov, E. V., Golovin, D., et al. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2488200 · Cited by 870
The master algorithm: how the quest for the ultimate learning machine will remake our world
(2016) · Choice Reviews Online · doi:10.5860/choice.194685 · Cited by 810
DeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238202 · Cited by 791
Evidence-Based Software Engineering and Systematic Reviews
Kitchenham, B. A., Budgen, D., & Brereton, P. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b19467 · Cited by 766
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Ghani, R., USA, K. 1. T. 1. A. S. I. C. O. K. D. A. D. M. 2. C. I., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Knowledge Discovery in Data, ACM Special Interest Group on Management of Data, & SIGs, A. (2013) · doi:10.1145/2487575 · Cited by 753
Collaborative filtering and deep learning based recommendation system for cold start items
Jian, W., He, J., Chen, K., Zhou, Y., & Tang, Z. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.09.040 · Cited by 713
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676
Larman, C. (2004) · Prentice Hall PTR eBooks · Cited by 628
A systematic review of machine learning techniques for software fault prediction
Malhotra, R. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2014.11.023 · Cited by 620
On testing non-testable programs
Weyuker, E. J. (1982) · The Computer Journal · doi:10.1093/comjnl/25.4.465 · Cited by 606
Metamorphic Testing: A New Approach for Generating Next Test CasesPreprint
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 603
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J. E., & Sculley, D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098043 · Cited by 585
Bakir, G. H., Hofmann, T., Schölkopf, B., Smola, A. J., Taskar, B., & Vishwanathan, S. V. N. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7443.001.0001 · Cited by 579
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sánchez, A. B., & Ruiz–Cortés, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 571
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562
An empirical analysis of flaky tests
Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014) · SIGSOFT FSE · doi:10.1145/2635868.2635920 · Cited by 551
Fink, O., Wang, Q., Svensén, M., Dersin, P., Lee, W.-J., & Ducoffe, M. (2020) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2020.103678 · Cited by 549
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. C. W. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330579 · Cited by 465
Single data extraction generated more errors than double data extraction in systematic reviews
Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., & Klassen, T. P. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.11.010 · Cited by 464
Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0
Bourque, P., & Fairley, R. E. (2014) · IEEE Computer Society Press eBooks · Cited by 455
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Fairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394
Establishing the rules for building trustworthy AI
Floridi, L. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0055-y · Cited by 394
Fairness-aware Learning through Regularization Approach
Kamishima, T., Akaho, S., & Sakuma, J. (2011) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2011.83 · Cited by 383
Anonymizing Classification Data for Privacy Preservation
Fung, B. C. M., Wang, K., & Yu, P. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.1015 · Cited by 351
Concolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238172 · Cited by 337
Learning under Concept Drift: an OverviewPreprint
Žliobaitė, I. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.4784 · Cited by 332
On the tradeoff between privacy and utility in data publishing
Li, T., & Li, N. (2009) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557079 · Cited by 328
A comprehensive study on deep learning bug characteristics
Islam, J., Nguyen, G., Pan, R., & Rajan, H. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338955 · Cited by 315
An empirical study on TensorFlow program bugs
Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213866 · Cited by 311
DLFuzz: differential fuzzing testing of deep learning systems
Guo, J., Jiang, Y., Zhao, Y., Chen, Q., & Sun, J. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264835 · Cited by 311
Azeem, M. I., Palomba, F., Shi, L., & Wang, Q. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.009 · Cited by 307
Naming the pain in requirements engineering
Méndez, D., Wagner, S., Kalinowski, M., Felderer, M., Mafra, P., Vetrò, A., Conte, T. U., Christiansson, M.-T., Greer, D., Lassenius, C., et al. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9451-7 · Cited by 303
Taxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380395 · Cited by 299
Identifying quality-requirement conflicts
BOEHM, B. W., & In, H. P. (1996) · IEEE Software · doi:10.1109/52.506460 · Cited by 295
Efficient memory-based learning for robot control
Moore, A. (2021) · doi:10.48456/tr-209 · Cited by 284
DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks
Feng, Y., Shi, Q., Gao, X., Wan, J., Fang, C., & Chen, Z. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397357 · Cited by 257
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256
Testing machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
Pospieszny, P., Czarnacka-Chrobot, B., & Kobylinski, A. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2017.11.066 · Cited by 247
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Large-scale machine learning at twitter
Lin, J., & Kolcz, A. (2012) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2213836.2213958 · Cited by 231
On testing machine learning programs
Braiek, H. B., & Khomh, F. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110542 · Cited by 226
Machine Learning: The High Interest Credit Card of Technical Debt
Sculley, D., Holt, G. D., Golovin, D., Davydov, E. V., Phillips, T., Ebner, D., Chaudhary, V., & Young, M. (2014) · Cited by 218
MODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W., Zhang, X., & Grama, A. Y. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236082 · Cited by 217
Data Lifecycle Challenges in Production Machine Learning: A Survey
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018) · ACM SIGMOD Record · doi:10.1145/3299887.3299891 · Cited by 214
Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., & Crnkovic, I. (2019) · Lecture Notes in Business Information Processing · doi:10.1007/978-3-030-19034-7_14 · Cited by 214
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210
Machine learning applied to software testing: A systematic mapping study
Durelli, V. H. S., Durelli, R. S., Borges, S. S., Endo, A. T., Eler, M. M., Dias, D. R. C., & Guimaraes, M. P. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2019.2892517 · Cited by 207
Automated directed fairness testing
Udeshi, S., Arora, P., & Chattopadhyay, S. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238165 · Cited by 200
Data scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
Alexander, I. F., & Stevens, R. (2002) · Cited by 191
Dwarakanath, A., Ahuja, M., Sikand, S., Rao, R. M., Bose, R. P. J. C., Dubash, N., & Podder, S. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213858 · Cited by 191
Λουρίδας, Π., & Ebert, C. (2016) · IEEE Software · doi:10.1109/ms.2016.114 · Cited by 190
Cognitive Biases in Software Engineering: A Systematic Mapping Study
Mohanani, R., Salman, I., Turhan, B., Rodríguez, P., & Ralph, P. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877759 · Cited by 185
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184
When and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182
Black box fairness testing of machine learning models
Aggarwal, A., Lohia, P., Nagar, S., Dey, K., & Saha, D. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338937 · Cited by 180
Misbehaviour prediction for autonomous driving systems
Stocco, A., Weiß, M., Calzana, M., & Tonella, P. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380353 · Cited by 176
Gade, K., Geyik, S. C., Kenthapadi, K., Mithal, V., & Taly, A. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3332281 · Cited by 171
Bell, J., Legunsen, O., Hilton, M. L., Eloussi, L., Yung, T., & Marinov, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180164 · Cited by 170
"Why Software Is Eating the World"
Zuckerberg, M. (2011) · UWM Digital Commons (University of Wisconsin–Milwaukee) · Cited by 169
Fairway: a way to build fair ML software
Chakraborty, J., Majumder, S., Yu, Z., & Menzies, T. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409697 · Cited by 167
Deep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167
DeepStellar: model-based quantitative analysis of stateful deep learning systems
Du, X., Xie, X., Li, Y., Ma, L., Liu, Y., & Zhao, J. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338954 · Cited by 166
Is neuron coverage a meaningful measure for testing deep neural networks?
Harel-Canada, F., Wang, L., Gulzar, M. A., Gu, Q., & Kim, M. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409754 · Cited by 166
Metamorphic testing for software quality assessment: A study of search engines
Zhou, Z. Q., Xiang, S., & Chen, T. Y. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2478001 · Cited by 163
CONSTRUE/TIS: A System for Content-Based Indexing of a Database of News Stories
Hayes, P. J., & Weinstein, S. P. (1990) · Innovative Applications of Artificial Intelligence · Cited by 159
Explainable software analytics
Dam, H. K., Tran, T., & Ghose, A. (2018) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3183399.3183424 · Cited by 158
Vartak, M., Subramanyam, H., Lee, W.-E., Viswanathan, S., Husnoo, S., Madden, S. R., & Zaharia, M. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939516 · Cited by 157
Model-based exploration of the frontier of behaviours for deep learning system testing
Riccio, V., & Tonella, P. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409730 · Cited by 152
Testing scientific software: A systematic literature review
Kanewala, U., & Bieman, J. M. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.05.006 · Cited by 150
Supervised Machine Learning Applied to Link Prediction in Bipartite Social Networks
Benchettara, N., Kanawati, R., & Rouveirol, C. (2010) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2010.87 · Cited by 148
An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
Estimating GPU memory consumption of deep learning models
Gao, Y., Liu, Y., Zhang, H., Li, Z., Zhu, Y., Lin, H., & Yang, M. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417050 · Cited by 147
Software reuse strategies and component markets
Ravichandran, T., & Rothenberger, M. A. (2003) · Communications of the ACM · doi:10.1145/859670.859678 · Cited by 146
The Practitioner's Guide to Data Quality Improvement
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-17212-4 · Cited by 143
A comprehensive study of autonomous vehicle bugs
Garcia, J., Feng, Y., Shen, J., et al. (2020) · doi:10.1145/3377811.3380397 · Cited by 140
On Challenges in Machine Learning Model Management
Schelter, S., Bießmann, F., Januschowski, T., Salinas, D., Seufert, S., & Szarvas, G. (2015) · IEEE Data(base) Engineering Bulletin · Cited by 139
Generating effective test cases for self-driving cars from police reports
Gambi, A., Huynh, T., & Fraser, G. (2019) · ESEC/SIGSOFT FSE · doi:10.1145/3338906.3338942 · Cited by 138
Zhang, J. M., Zhang, L., Harman, M., Hao, D., Jia, Y., & Zhang, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2809496 · Cited by 137
Requirements-driven test generation for autonomous vehicles with machine learning components
Tuncali, C. E., Fainekos, G., Prokhorov, D., Ito, H., & Kapinski, J. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2955903 · Cited by 129
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126
Neural network models for software development effort estimation: a comparative study
Nassif, A. B., Azzeh, M., Capretz, L. F., & Ho, D. (2016) · Neural Computing and Applications · doi:10.1007/s00521-015-2127-1 · Cited by 122
A comprehensive study on challenges in deploying deep learning based software
Chen, Z., Cao, Y., Liu, Y., Wang, H., Xie, T., & Liu, X. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409759 · Cited by 122
An experience report on defect modelling in practice
Tantithamthavorn, C. K., & Hassan, A. E. (2018) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3183519.3183547 · Cited by 121
Practical Software Requirements: A Manual of Content and Style
Kovitz, B. (1998) · Cited by 119
Aligning software engineering education with industrial needs: A meta-analysis
Garousi, V., Giray, G., Tüzün, E., Catal, C., & Felderer, M. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.06.044 · Cited by 119
Self-citationConcept drift detection for streaming data
Wang, H., & Abraham, Z. (2015) · doi:10.1109/ijcnn.2015.7280398 · Cited by 116
Provenance Analysis for Missing Answers and Integrity Repairs.
Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., et al. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 112
Challenges in the Deployment and Operation of Machine Learning in Practice
Baier, L., Jöhren, F., & Seebacher, S. (2019) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000095028 · Cited by 110
Economic perspectives in test automation
Ramler, R., & Wolfmaier, K. (2006) · International Workshop on Automation of Software Test (AST) · doi:10.1145/1138929.1138946 · Cited by 106
Boosting operational DNN testing efficiency through conditioning
Li, Z., Ma, X., Xu, C., Cao, C., Xu, J., & Lü, J. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338930 · Cited by 105
ModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI
Hummer, W., Muthusamy, V., Rausch, T., Dube, P., El Maghraoui, K., Murthi, A., & Oum, P. (2019) · 2019 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2019.00025 · Cited by 105
Mitchell, P. (2011) · Current Medical Research and Opinion · doi:10.1185/03007995.2011.585394 · Cited by 102
On challenges in engineering IoT software systems
Motta, R. C., de Oliveira, K. M., & Travassos, G. H. (2018) · Brazilian Symposium on Software Engineering · doi:10.1145/3266237.3266263 · Cited by 100
What Is Really Different in Engineering AI-Enabled Systems?
Özkaya, İ. (2020) · IEEE Software · doi:10.1109/ms.2020.2993662 · Cited by 100
Practical Accuracy Estimation for Efficient Deep Neural Network Testing
Chen, J., Wu, Z. J., Wang, Z., You, H., Zhang, L., & Yan, M. X. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3394112 · Cited by 97
Software engineering for machine-learning applications: The road ahead
Khomh, F., Adams, B., Cheng, J., Fokaefs, M., & Antoniol, G. (2018) · IEEE Software · doi:10.1109/ms.2018.3571224 · Cited by 95
Studying Software Engineering Patterns for Designing Machine Learning Systems
Washizaki, H., Uchida, H., Khomh, F., & Guéhéneuc, Y. (2019) · International Workshop on Empirical Software Engineering in Practice (IWESEP) · doi:10.1109/iwesep49350.2019.00017 · Cited by 94
Testing embedded software: A survey of the literature
Garousi, V., Felderer, M., Karapıçak, Ç. M., & Yılmaz, U. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.06.016 · Cited by 90
Understanding Development Process of Machine Learning Systems: Challenges and Solutions
Nascimento, E., Ahmed, I., de Oliveira, E. C. C., Palheta, M. P., Steinmacher, I., & Conte, T. U. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870157 · Cited by 89
Engineering problems in machine learning systems
Kuwajima, H., Yasuoka, H., & Nakae, T. (2020) · Machine Learning · doi:10.1007/s10994-020-05872-w · Cited by 89
Correlations between deep neural network model coverage criteria and model quality
Yan, S., Tao, G., Liu, X., Zhai, J., Ma, S., Xu, L., & Zhang, X. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409671 · Cited by 84
Creating a Software Engineering Culture
Wiegers, K. E. (1996) · Cited by 83
Performance and fault-tolerance of neural networks for optimization
Protzel, P., Palumbo, D., & Arras, M. K. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.238315 · Cited by 82
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81
Detecting flaky tests in probabilistic and machine learning applications
Dutta, S., Shi, A., Choudhary, R., Zhang, Z., Jain, A., & Misailovíc, S. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397366 · Cited by 79
Nascimento, E., Nguyen‐Duc, A., Sundbø, I., & Conte, T. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03751 · Cited by 78
More About Software Requirements: Thorny Issues and Practical Advice
Wiegers, K. E. (2005) · Cited by 76
Requirements Engineering in the Days of Artificial Intelligence
Dalpiaz, F., & Niu, N. (2020) · IEEE Software · doi:10.1109/ms.2020.2986047 · Cited by 72
A Survey of Software Quality for Machine Learning Applications
Masuda, S., Ono, K., Yasue, T., & Hosokawa, N. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00061 · Cited by 71
Peng, Z. J., Yang, J., Chen, T.-H. P., & Ma, L. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417063 · Cited by 71
Groce, A., Kulesza, T., Zhang, C., Shamasunder, S., Burnett, M., Wong, W., Stumpf, S., Das, S., Shinsel, A., Bice, F., et al. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.59 · Cited by 69
Engineering AI Systems: A Research AgendaPreprint
Bosch, J., Crnković, I., & Olsson, H. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07522 · Cited by 68
A complexity theory viewpoint on the software development process and situational context
Clarke, P. M., O’Connor, R. V., & Leavy, B. (2016) · International Conference on Software and System Process · doi:10.1145/2904354.2904369 · Cited by 67
Just Enough Software Test Automation
Mosley, D. J., & Posey, B. (2002) · International Linear Collider · Cited by 66
ModelHub: Deep Learning Lifecycle Management
Miao, H., Li, A., Davis, L. S., & Deshpande, A. (2017) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2017.192 · Cited by 63
Artificial Intelligence and Software Engineering: Status and Future Trends.
Rech, J., & Althoff, K. (2004) · Cited by 62
An Approach to Software Testing of Machine Learning Applications
Murphy, C., Kaiser, G. E., & Arias, M. (2007) · doi:10.7916/d8r49znr · Cited by 61
Searching for build debt: experiences managing technical debt at Google
Morgenthaler, J. D., Gridnev, M., Sauciuc, R., & Bhansali, S. (2012) · doi:10.5555/2666036.2666037 · Cited by 60
Testing and quality validation for AI software–perspectives, issues, and practices
Tao, C., Gao, J., & Wang, T. (2019) · IEEE Access · doi:10.1109/access.2019.2937107 · Cited by 60
Towards Guidelines for Assessing Qualities of Machine Learning Systems
Siebert, J., Jöckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., & Aoyama, M. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-58793-2_2 · Cited by 60
The sensemaking-coevolution-implementation theory of software design
Ralph, P. (2015) · Science of Computer Programming · doi:10.1016/j.scico.2014.11.007 · Cited by 60
Is using deep learning frameworks free?
Liu, J., Huang, Q., Xia, X., Shihab, E., Lo, D., & Li, S. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1145/3377815.3381377 · Cited by 55
MODELDB: A System for Machine Learning Model Management.
Vartak, M. (2017) · Conference on Innovative Data Systems Research · Cited by 54
Teaching software engineering for AI-enabled systems
Kästner, C., & Kang, E. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3377814.3381714 · Cited by 53
Tata, S., Popescul, A., Najork, M., Colagrosso, M., Gibbons, J., Green, A. I., Mah, A., Smith, M. D., Garg, D., Meyer, C., et al. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098048 · Cited by 52
Catching up with Method and Process Practice: An Industry-Informed Baseline for Researchers
Klünder, J., Hebig, R., Tell, P., Kuhrmann, M., Nabende, J. N., Heldal, R., Krusche, S., Fazal-Baqaie, M., Felderer, M., Bocco, M. G., et al. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00036 · Cited by 50
Deep Convolutional Neural Network Design PatternsPreprint
Smith, L. N., & Topin, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00847 · Cited by 47
Kang, S., FELDT, R. H., & Yoo, S. (2020) · doi:10.1145/3387940.3391456 · Cited by 46
The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development
Smith, M. J., Sala, C., Kanter, J. M., & Veeramachaneni, K. K. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 46
Towards a Software Engineering Process for Developing Data-Driven Applications
Hesenius, M., Schwenzfeier, N., Meyer, O., et al. (2019) · doi:10.1109/raise.2019.00014 · Cited by 45
Sutton, C. A., Hobson, T., Geddes, J., et al. (2018) · ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) · doi:10.1145/3219819.3220057 · Cited by 44
Hulten, G. (2018) · Apress eBooks · doi:10.1007/978-1-4842-3432-7 · Cited by 43
Menzies, T. (2020) · IEEE Software · doi:10.1109/ms.2019.2954841 · Cited by 43
Moreb, M., Mohammed, T. A., & Bayat, O. (2020) · IEEE Access · doi:10.1109/access.2020.2970178 · Cited by 39
Think Your Artificial Intelligence Software Is Fair? Think Again
Bellamy, R., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Mehta, S., Mojsilović, A., Nagar, S., et al. (2019) · IEEE Software · doi:10.1109/ms.2019.2908514 · Cited by 37
Manifold for machine learning assurance
Byun, T., & Rayadurgam, S. (2020) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3377816.3381734 · Cited by 34
Privacy and anonymization for very large datasets
Muntés-Mulero, V., & Nin, J. (2009) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1645953.1646333 · Cited by 32
The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development
Smith, M. J., Sala, C., Kanter, J. M., et al. (2020) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/3318464.3386146 · Cited by 31
Machine Learning Software Engineering in Practice: An Industrial Case StudyPreprint
Rahman, M. S., Rivera, E., Khomh, F., Guéhéneuc, Y., & Lehnert, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07154 · Cited by 31
Embrace the challenges: software engineering in a big data world
Anderson, K. M. (2015) · International Workshop on BIG Data Software Engineering · doi:10.5555/2819289.2819297 · Cited by 30
Jung, J. (2005) · Cited by 28
Bridging the gap between ML solutions and their business requirements using feature interactions
Barash, G., Farchi, E., Jayaraman, I., et al. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3340442 · Cited by 28
Operational calibration: debugging confidence errors for DNNs in the field
Li, Z., Ma, X., Xu, C., Xu, J., Cao, C., & Lü, J. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409696 · Cited by 28
CERSE - Catalog for empirical research in software engineering: A Systematic mapping study
Molléri, J. S., Petersen, K., & Mendes, E. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.008 · Cited by 26
On Testing Data-Intensive Software Systems
Felderer, M., Russo, B., & Auer, F. (2019) · Springer eBooks · doi:10.1007/978-3-030-25312-7_6 · Cited by 26
Detecting and understanding real-world differential performance bugs in machine learning libraries
Tizpaz-Niari, S., Černý, P., & Trivedi, A. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3404540 · Cited by 26
Research works printed but not linked: 6
- Fujii, 2020, Guidelines for quality assurance of machine learning-based artificial intelligence, Int. J. Softw. Eng. Knowl. Eng., 1
- Wan, 2019, How does machine learning change software development practices?, IEEE Trans. Softw. Eng.
- Fei-Fei, L., 2010. ImageNet: crowdsourcing, benchmarking & other cool things. In: CMU VASC Seminar (Vol. 16). March. pp. 18-25.
- Schwerdtner, P., Grezner, F., Kapoor, N., Assion, F., Sass, R., Günther, W., Hüger, F., Schlicht, P., 2020. Risk assessment for machine learning models. In: Machine Learning for Autonomous Driving Workshop at the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R., 2014. Intriguing properties of neural networks. In: 2nd International Conference on Learning Representations (ICLR).
- Zaharia, 2018, Accelerating the machine learning lifecycle with mlflow, IEEE Data Eng. Bull., 41, 39
Cited by
Sorted by citations
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34
Metadata
- Authors
- Giray, G.
- Corresponding author
- Giray, Görkem
- ORCID
- Giray, Görkem
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2021-10
- Volume
- 180
- Pages
- 111031
- ISSN
- 0164-1212 (ISSN-L)
- OpenAlex
- W3112475255
- Cited by
- 187
- References
- 256
- Citations by year
- 2026: 23 · 2025: 37 · 2024: 38 · 2023: 43 · 2022: 36 · 2021: 10
- FWCI
- 28.71
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Journal of Systems and Software (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- software engineering for machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
