A Survey of Data Quality Requirements That Matter in ML Development Pipelines

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The UK government has recently published an Algorithmic Transparency Standard alongside templates designed to help public sector organisations document the datasets that underlie their ML tools.

“The UK government has recently published an Algorithmic Transparency Standard 3 alongside templates designed to help public sector organisations to document the datasets that underlie their ML tools.”

• cites [3] 3. Lora Aroyo, Matthew Lease, Praveen Paritosh, and Mike Schaekermann. 2022. Data excellence for AI: why should you care? Interactions 29, 2 (2022), 66-69. sourceCould not verify

  The Discussion section raises general questions about dataset documentation, but it does not mention a UK government standard or templates for public-sector organisations. Searches for the Algorithmic Transparency Standard and for government templates found no passage in our copy addressing their publication or purpose; page numbers were not available. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Standards of this kind crystallise an ever growing corpus of academic literature that has explored ML data quality challenges and ways to mitigate them.

“Standards of this kind crystallise an ever growing corpus of academic literature that has explored ML data quality challenges and ways to mitigate them [25,57,64,70].”

• cites [25] 25. Fernando Gualo, Moisés Rodríguez, Javier Verdugo, Ismael Caballero, and Mario Piattini. 2021. Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software 176 (2021), 110938. sourcePartly supported

  Section 5.1 (page not provided) says ISO/IEC 25012 and 25024 establish a common understanding of data quality through international expert agreement. The Introduction describes standards-based methods for evaluating data quality. Neither passage says the standards crystallise a growing academic literature on ML-specific data quality challenges and mitigations. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [57] 57. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22 (2021). sourceNot checked — no full text of the source
• cites [64] 64. Annabel Rothschild, Justin Booker, Christa Davoll, Jessica Hill, Venise Ivey, Carl DiSalvo, Ben Rydal Shapiro, and Betsy DiSalvo. 2022. Towards fair and pro-social employment of digital pieceworkers for sourcing machine learning training data. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1-9. sourceNot checked — no full text of the source
• cites [70] 70. Ikbal Taleb, Mohamed Adel Serhani, Chafik Bouhaddioui, and Rachida Dssouli. 2021. Big data quality framework: a holistic approach to continuous quality management. Journal of Big Data 8, 1 (2021), 1-41. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The topic of data quality is beginning to transcend beyond data management and into HCI, which accommodates holistic considerations such as how people search for relevant datasets, how developers perceive data work, and best ways of using crowdsourcing to generate, evaluate or label data.

“Because of this, the topic of data quality is beginning to transcend beyond the field of data management and into the realm of Human Computer Interaction (HCI), which accommodates holistic considerations such as how people search for relevant datasets [42], how developers perceive data work [64] and the best ways of using crowdsourcing to generate, evaluate or label data [71].”

• cites [42] 42. Sunho Kim, Ricardo Pérez-Castillo, Ismael Caballero, and Downgwoo Lee. 2022. Organizational process maturity model for IoT data quality management. Journal of Industrial Information Integration 26 (2022), 100256. sourceNot checked — no full text of the source
• cites [64] 64. Annabel Rothschild, Justin Booker, Christa Davoll, Jessica Hill, Venise Ivey, Carl DiSalvo, Ben Rydal Shapiro, and Betsy DiSalvo. 2022. Towards fair and pro-social employment of digital pieceworkers for sourcing machine learning training data. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1-9. sourceNot checked — no full text of the source
• cites [71] 71. Linnet Taylor. 2017. What is data justice? The case for connecting digital rights and freedoms globally. Big Data & Society 4, 2 (2017), 2053951717736335. sourceCould not verify

  Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

New standards are currently being developed in industry to guide businesses on how to define, implement and measure data quality throughout the ML development lifecycle.

“Similar trends are happening in industry, where new standards are currently being developed to guide businesses on how to define, implement and measure data quality throughout the ML development lifecycle [18].”

• cites [18] 18. Samuel Budd, Emma C Robinson, and Bernhard Kainz. 2021. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical Image Analysis 71 (2021), 102062. sourceCould not verify

  Section 4, “Practical Considerations” (p. 11), discusses developing and deploying deep-learning applications, while Section 5 (p. 15) calls for baseline methods to assess their performance. Neither passage addresses industry standards for defining, implementing, or measuring data quality across the ML development lifecycle; searches for “industry standards” and “data quality” found no passage that does. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

While the role of dynamic processes and multi-stakeholder configurations is increasingly recognised by data practitioners, it is less clear how traditional data quality frameworks and notions of data accountability are adapting to ML development pipelines.

“While the role of these dynamic processes and multi-stakeholder configurations is increasingly being recognised by data practitioners, it is less clear how traditional data quality frameworks and notions of data accountability are adapting to ML development pipelines [35].”

• cites [35] 35. Sara Hooker. 2021. Moving beyond "algorithmic bias is a data problem". Patterns 2, 4 (2021), 100241. sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Multi-dataset-multi-model pipelines, where existing ML models are used for pre-processing data or training new ML models, are becoming increasingly common.

“Other scenarios that are becoming increasingly common are multi-dataset-multi-model pipelines, where existing ML models are used for pre-processing data or training new ML models [5].”

• cites [5] 5. Rob Ashmore, Radu Calinescu, and Colin Paterson. 2021. Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ACM Computing Surveys (CSUR) 54, 5 (2021), 1-39. sourcePartly supported

  Section 5.2.4 says an existing model can provide a starting point for training another, and Section 2 discusses preprocessing data collected from multiple sources. Neither passage says that existing ML models perform the preprocessing or that multi-dataset, multi-model pipelines are becoming increasingly common. Page numbers were not provided in our copy. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

The literature review was conducted using a systematic mapping protocol to select a small set of relevant articles from the larger literature at the intersection of data quality and ML.

“Our literature review was conducted using a systematic mapping protocol [54] in order to select a small set of relevant articles from the much larger collection of literature emerging at the intersection of data quality and ML.”

• cites [54] 54. Andrei Paleyes, Raoul-Gabriel Urma, and Neil D Lawrence. 2022. Challenges in deploying machine learning: a survey of case studies. Comput. Surveys 55, 6 (2022), 1-29. sourceCould not verify

  The Introduction describes the kinds of ML deployment papers included in Paleyes et al.'s survey and says it generally considered papers from the preceding six years. It does not describe a systematic mapping protocol for selecting articles at the intersection of data quality and ML (searched for “systematic mapping” and literature-selection criteria). The citing authors' own review process cannot be verified from this source. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Providing examples of what requirements can look like helps to guide practitioners in clarifying their own rules.

“The authors found that providing examples of what the requirements can look like helps to guide practitioners in clarifying their own rules.”

• cites [24] 24. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86-92. sourcePartly supported

  In Questions and Workflow (p. 4), the authors provide an example datasheet to illustrate how their questions might be answered; Objectives (p. 2) says the questions are nonprescriptive and should vary by context. These passages support the idea of offering adaptable examples, but the paper does not report finding that examples help practitioners clarify their own rules. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Merino et al. draw on Wang and Strong's distinction between intrinsic, contextual, accessibility, and representational aspects when using ISO/IEC 25012 and ISO/IEC 25024.

“Merino et al. [47] draw on Wang and Strong [74]'s canonical distinction between the intrinsic, contextual, accessibility, and representational aspects of datasets when using the above mentioned industry standards ISO/IEC 25012 and ISO/IEC 25024.”

• cites [47] 47. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017). sourceCould not verify

  The cited paper’s Introduction (p. 1) and optimization discussion (p. 3) concern adversarial robustness, not dataset-quality standards. Searches for Merino, Wang and Strong, ISO/IEC 25012 and 25024, and the four named quality aspects found no passage addressing the claim. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [74] 74. April Yi Wang, Dakuo Wang, Jaimie Drozdal, Xuye Liu, Soya Park, Steve Oney, and Christopher Brooks. 2021. What makes a well-documented notebook? a case study of data scientists' documentation practices in kaggle. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-7. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

Data quality must be defined using stage-specific approaches that are sensitive to where in the ML lifecycle the data are encountered and who encounters them.

“Instead, data quality must be defined using stage-specific approaches that are sensitive to where in the ML lifecycle the data are encountered and who encounters them [25,57].”

• cites [25] 25. Fernando Gualo, Moisés Rodríguez, Javier Verdugo, Ismael Caballero, and Mario Piattini. 2021. Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software 176 (2021), 110938. sourcePartly supported

  The Introduction (page unavailable) says data should have an appropriate level of quality for its present and future use. Section 2 (page unavailable) describes selecting quality characteristics to fit an organization’s requirements, with input from managerial and technical stakeholders. These passages support tailoring data-quality evaluation to its purpose, but they do not say it must be defined by ML lifecycle stage or by who encounters the data at each stage. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [57] 57. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22 (2021). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The data quality metrics that matter most in classification tasks are proposed to include class overlap, outliers, boundary complexity, label noise, and class imbalance.

“For example, the data quality metrics that matter most in classification tasks are proposed to include class overlap, outliers, boundary complexity, label noise, and class imbalance.”

• cites [25] 25. Fernando Gualo, Moisés Rodríguez, Javier Verdugo, Ismael Caballero, and Mario Piattini. 2021. Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software 176 (2021), 110938. sourceCould not verify

  Section 2.1 discusses ISO/IEC data-quality characteristics, and the Additional information section lists properties such as completeness, consistency, and accuracy. Neither those passages nor searches for class overlap, outliers, boundary complexity, label noise, and class imbalance found a proposal of metrics for classification tasks in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Datasets will require governance standards and specifications to support their maintenance, especially in larger organisations that handle multiple datasets.

“Datasets will require governance standards and specifications to support their maintenance, especially in larger organisations that handle multiple datasets [25].”

• cites [25] 25. Fernando Gualo, Moisés Rodríguez, Javier Verdugo, Ismael Caballero, and Mario Piattini. 2021. Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software 176 (2021), 110938. sourcePartly supported

  The Conclusions (page not supplied) say that frameworks based on ISO/IEC standards are needed to evaluate data quality over time. Section 4.3 also recommends updating repository documentation as business rules are refined. The paper does not establish that dataset governance standards are required for maintenance, or that the need is especially acute in larger organizations handling multiple datasets. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

Documentation about data collection methods provides the information needed to make informed decisions about using a dataset and to avoid unintentional misuse.

“For consumers, documentation about data collection methods provides the information needed to make informed decisions about using a dataset and to avoid unintentional misuse [23,25].”

• cites [23] 23. Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. From data mining to knowledge discovery in databases. AI magazine 17, 3 (1996), 37-37. sourceNot checked — no full text of the source
• cites [25] 25. Fernando Gualo, Moisés Rodríguez, Javier Verdugo, Ismael Caballero, and Mario Piattini. 2021. Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software 176 (2021), 110938. sourceCould not verify

  Section 2 describes a document of business rules used by an evaluation team, not documentation of data collection methods for dataset consumers. Searches for data collection, consumer decisions, and misuse found no passage saying such documentation helps consumers decide whether to use a dataset or avoid misuse. Page numbers were not available in our copy. (LLM, conf=0.73)

  Experimental — accuracy not yet measured

May combine two claims.

Model inspection and visualisation methods can support the interpretability of the model, which can in turn influence its perceived quality.

“Besides performance, model inspection and visualisation methods can also support the interpretability of the model, which can in turn influence its perceived quality [4].”

• cites [4] 4. Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion 58 (2020), 82-115. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Part of supporting stakeholder self-organisation into a robust data quality workflow is about providing incentives to individuals and organisations.

“Part of that is about providing incentives to individuals and organisations [23,53].”

• cites [23] 23. Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. From data mining to knowledge discovery in databases. AI magazine 17, 3 (1996), 37-37.Not checked — no full text of the source
• cites [53] 53. Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. 2020. Bias in data-driven artificial intelligence systems-An introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10, 3 (2020), e1356. sourceCould not verify

  The sections on bias-aware data collection and future directions discuss data quality, collaboration, and bias mitigation, but do not address providing incentives to individuals or organisations as part of a data-quality workflow. Searches for “incentives” and for motivating stakeholders to improve data quality found no passage making that point. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

May combine two claims.

At present, the field of ML suffers from the devaluation of data work, with model development tasks being held in higher esteem than data quality processes.

“At present, the field of ML suffers from the devaluation of data work, with model development tasks being held in higher esteem than data quality processes [35,64].”

• cites [35] 35. Sara Hooker. 2021. Moving beyond "algorithmic bias is a data problem". Patterns 2, 4 (2021), 100241. sourceCould not verify

  The opening section discusses the costs of fixing data bias, and 'Why a more nuanced discussion of the origins of bias matters' argues for considering both data and model design. Neither addresses whether ML researchers hold model development in higher esteem than data-quality work. Searches for devaluation of data work and research incentives found no passage making that comparison. (LLM, conf=0.75)

  Experimental — accuracy not yet measured
• cites [64] 64. Annabel Rothschild, Justin Booker, Christa Davoll, Jessica Hill, Venise Ivey, Carl DiSalvo, Ben Rydal Shapiro, and Betsy DiSalvo. 2022. Towards fair and pro-social employment of digital pieceworkers for sourcing machine learning training data. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1-9. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Practical approaches to professionalising data work include establishing membership organisations and review panels with standardised codes of conduct.

“Practical approaches to this include establishing membership organisations and review panels with standardised codes of conduct [36].”

• cites [36] 36. Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 560-575. sourcePartly supported

  Section 4.3, “Dataset Audits and Reviews” (page not available), advocates reviews throughout the dataset lifecycle, supporting the review component of the claim. The passages do not propose establishing membership organisations or review panels with standardised codes of conduct. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Many selected papers discussed topics that went beyond the original focus on data quality in the technical sense.

“Besides extending into practical techniques, many of our selected papers discussed topics that went beyond our original focus on data quality in the technical sense.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There is significant overlap between these fields, which the authors attempt to illustrate in Figure 2.

“Nonetheless there is significant overlap which we attempt to illustrate in Figure 2.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Thematic coding of the selected articles revealed additional themes related to the development stages of ML and the scope occupied by data quality management in the wider ML literature.

“We analysed the selected articles using thematic coding, which revealed additional themes related to the development stages of ML and the scope occupied by data quality management in the wider ML literature.”

Classified as probably not a substantial claim.

Some authors' comments and data quality requirements were targeted to specific stages in the ML development pipeline.

“Once we started reading the papers, we found that some of the authors' comments and data quality requirements were targeted to specific stages in the ML development pipeline.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The precise data quality requirements will not be known upfront, and new use cases may emerge as the model matures.

“It is common that the precise data quality requirements will not be known upfront, and new use cases may emerge as the model matures.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Accessibility requirements include the quality and security of infrastructure used for data storage, access and maintenance, which must be in place before data are available to develop ML models.

“Lastly, requirements around accessibility include the quality and security of infrastructure used for data storage, access and maintenance, which must be in place before the data are available to develop ML models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

During later stages of ML deployment, intrinsic data quality tasks shift from working with training data to preparation of serving data received from the outside world.

“During the later stages of ML deployment, the tasks of intrinsic data quality shift from working with training data to the preparation of serving data received from the outside world.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

A large part of the representational aspect of data quality involves documenting how the above requirements were met.

“A large part of the representational aspect of data quality involves documenting how the above requirements were met.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Intrinsic data quality requirements initially target the data collection stage, where careful monitoring and human-in-the-loop methods can support acquisition of data that are accurate, reliably sourced and clean from the outset.

“Requirements around intrinsic data quality may initially be targeted to the data collection stage, where careful monitoring and human-in-the-loop methods can support the acquisition of data that are accurate, reliably sourced and clean from the outset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Contextual requirements during data collection, especially in sociocultural contexts, include compliance with ethical and inclusivity guidelines.

“Other contextual requirements during data collection, especially in sociocultural contexts, relate to compliance with ethical and inclusivity guidelines.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In early stages of ML development, contextual fit of a dataset may be improved through feature selection and re-sampling of the training distribution.

“When approaching the early stages of ML development, the contextual fit of the dataset may be improved through steps such as feature selection and re-sampling of the training distribution.”

Classified as probably not a substantial claim.

Contextual data quality requires understanding the anticipated ML use case and characteristics of the end users before data collection.

“With contextual data quality, the authors in our review highlighted the importance of understanding the anticipated ML use case and characteristics of the end users before data collection.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Data quality frameworks and standards are accompanied by a growing range of tools for data pre-processing, documentation and assurance.

“These are accompanied by a growing range of tools for data pre-processing, documentation and assurance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Performance metrics need to be tailored to the specific use case of the model.

“One of the themes emerging from the above authors is that performance metrics need to be tailored to the specific use case of the model.”

Classified as probably not a substantial claim.

There is value in simplified data quality frameworks that are tailored to specific use cases or technologies.

“It was also found that there is value in simplified data quality frameworks that are tailored to specific use cases or technologies [41].”

• cites [41] 41. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Towards proving the adversarial robustness of deep neural networks. arXiv preprint arXiv:1709.02802 (2017). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Small contextually relevant datasets can lead to better and fairer performance than large data.

“Small contextually relevant datasets can lead to better and fairer performance than large data.”

Probably not a substantial claim: may not pass S4 (based).

Data quality frameworks and standards present practitioners with dozens of possible criteria to comply with.

“Earlier in this paper we noted that data quality frameworks and standards present practitioners with dozens of possible criteria to comply with.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ISO has begun to develop the ISO/IEC 5259 standard which focuses on data quality for the fields of analytics and ML, as well as ISO/IEC DIS 8183 that addresses the AI data life cycle framework.

“More recently, ISO has begun to develop the ISO/IEC 5259 standard which focuses on data quality for the fields of analytics and ML, as well as ISO/IEC DIS 8183 that addresses the AI data life cycle framework.”

• cites ISO/IEC 5259Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ISO/IEC DIS 8183Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Training data for ML algorithms can be collected in a variety of ways.

“Training data for ML algorithms can be collected in a variety of ways.”

• cites [61] 61. Jimmy Rising. 2002. Justice and Ethics. (2002).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Practitioners face a challenge of addressing new data quality requirements using frameworks and practices already familiar to the data management community.

“This raises a twofold challenge: 1) how to navigate the ML literature and select only those data quality requirements that are meaningful to the practitioner's use case, and 2) how to address the new requirements using frameworks and practices that are already familiar to the data management community.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Messy or inaccurate data can disturb the operational efficiency of businesses.

“Messy or inaccurate data can also disturb the operational efficiency of businesses, with estimates of 10% to 30% of revenue being spent on resolving data quality issues [30].”

• cites [30] 30. Haibo He and Edwardo A Garcia. 2009. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering 21, 9 (2009), 1263-1284. sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

A stage-specific approach helps practitioners plan their data quality tasks in a manner that meets the needs of the stakeholders who will encounter the dataset, whether data subjects, software developers or organisations.

“This helps practitioners to plan their data quality tasks in a manner that meets the needs of the stakeholders who will encounter the dataset, whether it be data subjects, software developers or organisations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These newer standards address processes that can be employed by various stakeholders at different stages of the AI life cycle, which differs from earlier data quality guidelines that tended to view quality as a uniform outcome that fulfils a pre-defined list of desired criteria.

“These newer standards address processes that can be employed by various stakeholders at different stages of the AI life cycle, which differs from earlier data quality guidelines that tended to view quality as a uniform outcome that fulfils a pre-defined list of desired criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These data quality models are complemented by standards which recognise that organisations differ in their preparedness to define and execute data quality assurance.

“These data quality models are complemented by standards which recognise that organisations differ in their preparedness to define and execute data quality assurance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ISO/IEC 25012 provides guidance on how to define the data quality characteristics that matter to an organisation.

“For example, the International Organization for Standardization (ISO) standard ISO/IEC 25012 provides guidance on how to define the data quality characteristics that matter to an organisation.”

• cites ISO/IEC 25012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Roh et al. group data collection methods for ML into three categories: data acquisition, data labelling, and improvement.

“In their comprehensive survey of data collection methods for ML, Roh et al. [61] group these into three categories: 1) data acquisition (including discovery, augmentation and generation), 2) data labelling (using manual or semi-supervised approaches) and 3) improvement (cleaning the data itself or improving the model built upon it).”

• cites [61] 61. Jimmy Rising. 2002. Justice and Ethics. (2002).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

ISO/IEC 25024 guides organisations in defining the data quality assurance criteria and ways of measuring them quantitatively.

“This latter task is addressed by the standard ISO/IEC 25024, which guides organisations in defining the data quality assurance criteria and ways of measuring them quantitatively.”

• cites ISO/IEC 25024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ISO 8000-61 specifies the pure activities of enhancing data quality processes, while ISO 8000-62 defines ways to assess the maturity, or readiness, of organisations to implement these data quality tasks.

“The ISO 8000-61 standard specifies the pure activities of enhancing data quality processes, while ISO 8000-62 defines ways to assess the maturity, or readiness, of organisations to implement these data quality tasks.”

• cites ISO 8000-61Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ISO 8000-62Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Industry researchers often rely on separate data collection, processing and storage systems that sit across multiple company functions, requiring formal data management guidelines to ensure consistency and coordination across teams.

“Industry researchers, however, often rely on separate data collection, processing and storage systems that sit across multiple company functions, requiring formal data management guidelines to ensure consistency and coordination across teams.”

• cites [52] 52. Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. 2012. A unifying view on dataset shift in classification. Pattern recognition 45, 1 (2012), 521-530. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Defining these characteristics is a pre-requisite to deciding how data quality can be evaluated in a practical sense.

“Defining these characteristics is a pre-requisite to deciding how data quality can be evaluated in a practical sense.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Estimates of 10% to 30% of revenue are spent on resolving data quality issues.

“Messy or inaccurate data can also disturb the operational efficiency of businesses, with estimates of 10% to 30% of revenue being spent on resolving data quality issues [30].”

• cites [30] 30. Haibo He and Edwardo A Garcia. 2009. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering 21, 9 (2009), 1263-1284. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Routines for ensuring transparency in ML datasets and ML development pipelines are being encouraged by national and international organisations such as the OECD and the Open Government Partnership.

“Routines for ensuring transparency in ML datasets and ML development pipelines are being encouraged by national and international organisations such as the OECD 1 and the Open Government Partnership 2 .”

• cites OECD 1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Open Government Partnership 2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Granular specifications of what makes a good quality dataset exist, with essential dimensions such as accuracy, completeness, consistency and validity being just some of the 60 dimensions identified in the wider data management literature.

“This has been accompanied by granular specifications of what makes a good quality dataset, with essential dimensions such as accuracy, completeness, consistency and validity being just some of the 60 dimensions identified in the wider data management literature [13].”

• cites [13] 13. A Black and P van Nederpelt. 2020. Dimensions of Data Quality (DDQ) Research Paper. http://www.dama-nl.org/wp- content/uploads/2020/09/DDQ-Dimensions-of-Data-Quality-Research-Paper-version-1.2-d.d.-3-Sept-2020.pdf. DAMA NL Foundation (2020). [Online; accessed 6/10/21].Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The extent to which these data collection methods are used varies depending on the use case and the type of data upon which an ML system relies.

“The extent to which these data collection methods are used varies depending on the use case and the type of data upon which an ML system relies.”

Probably not a substantial claim: may not pass S4 (based).

The growing range of ML data management guidelines, frameworks and standards presents practitioners with a vast range of possible criteria to aspire to, on top of traditional data management practices established in previous decades.

“The growing range of ML data management guidelines, frameworks and standards presents practitioners with a vast range of possible criteria to aspire to, on top of the traditional data management practices that were established in previous decades.”

Classified as probably not a substantial claim.

The growing use of ML across industries, and the high-stakes nature of some uses, is being accompanied by greater scrutiny of the processes that determine the output of ML-based decision-support systems.

“The growing use of ML across industries, and the high-stakes nature of some of the above uses [17,44], is being accompanied by greater scrutiny of the processes that determine the output of ML-based decision-support systems [51].”

• cites [17] 17. J. Data Inform. Quality, Vol. xx, No. x, Article xxx. Publication date: x 2023.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [44] 44. J. Data Inform. Quality, Vol. xx, No. x, Article xxx. Publication date: x 2023.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [51] 51. Tanushree Mitra, Clayton J Hutto, and Eric Gilbert. 2015. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1345-1354. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Longstanding definitions of data quality have viewed good quality data as 'data that are fit for use by data consumers'.

“Longstanding definitions of data quality have viewed good quality data as”

• cites [74] 74. April Yi Wang, Dakuo Wang, Jaimie Drozdal, Xuye Liu, Soya Park, Steve Oney, and Christopher Brooks. 2021. What makes a well-documented notebook? a case study of data scientists' documentation practices in kaggle. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-7. sourceNot checked — no full text of the source

May combine two claims.

Poor quality datasets and data science pipelines can compromise ML systems.

“Poor quality datasets and data science pipelines can compromise ML systems in a number of ways.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These standards are still under development, so there is value in publications that inform practitioners of how data quality applies to ML tasks.

“These standards are still under development, so there is value in publications that inform practitioners of how data quality applies to ML tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ML data tend to be managed differently depending on whether the system is within an academic or industry setting.

“It is worth noting that ML data tend to be managed differently depending on whether the system is within an academic or industry setting [52].”

• cites [52] 52. Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. 2012. A unifying view on dataset shift in classification. Pattern recognition 45, 1 (2012), 521-530. sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Practitioners face a challenge of navigating the ML literature and selecting only those data quality requirements that are meaningful to their use case.

“This raises a twofold challenge: 1) how to navigate the ML literature and select only those data quality requirements that are meaningful to the practitioner's use case, and 2) how to address the new requirements using frameworks and practices that are already familiar to the data management community.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There may be other venues with relevant contributions that were not included in the selection.

“We are aware that there may be other venues with relevant contributions that were not included in our selection.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The analysis was truncated after examining the first 30 Google Scholar results because many did not meet the inclusion criteria.

“We truncated our analysis after examining the first 30 results, as many of them did not meet our inclusion criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Standardisation helps engineers to ingest datasets from multiple sources and build interoperable solutions.

“This helps engineers to ingest datasets from multiple sources and build interoperable solutions.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The 32 papers reviewed are only a small part of the growing number of articles related to data quality in ML that exist in reality.

“While we sought to gather a representative sample of papers, it is important to acknowledge that the 32 papers reviewed here are only a small part of the growing number of articles related to data quality in ML that exist in reality.”

Classified as probably not a substantial claim.

The distinction between data quality planning and implementation informed the selection criteria of the review.

“This distinction between data quality planning and implementation informed the selection criteria of our review.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Kim et al. developed new frameworks for assessing and improving maturity of IoT data quality processes based on ISO 8000-61 and ISO 8000-62.

“Kim et al. [41], who developed new frameworks for assessing and improving the maturity of IoT data quality processes based on the standards ISO 8000-61 and ISO 8000-62.”

• cites [41] 41. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Towards proving the adversarial robustness of deep neural networks. arXiv preprint arXiv:1709.02802 (2017). sourceNot checked — no full text of the source

May combine two claims.

ML model quality is dependent on good quality training, testing or serving data, as well as metadata in the form of documentation.

“the inclusion of model and documentation quality helped to highlight the areas where ML model quality is dependent on good quality training, testing or serving data, as well as metadata in the form of documentation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The rise of big data and ML applications has been accompanied by calls for publishing datasets in an open manner, as well as secure access mechanisms for restricted datasets, so that their value can be realised.

“The rise of big data and ML applications in recent decades has been accompanied by calls for publishing datasets in an open manner, as well as secure access mechanisms for restricted datasets, so that their value can be realised [75].”

• cites [75] 75. Ding Wang, Shantanu Prabhat, and Nithya Sambasivan. 2022. Whose AI Dream? In search of the aspiration in data annotation.. In CHI Conference on Human Factors in Computing Systems. 1-16. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The list of 60 dimensions by Black and van Nederpelt includes qualities related to data accuracy, lineage, currency, coverage, legal compliance and usability.

“the list of 60 dimensions created by Black and van Nederpelt [13] includes qualities related to data accuracy, lineage, currency, coverage, legal compliance and usability.”

• cites [13] 13. A Black and P van Nederpelt. 2020. Dimensions of Data Quality (DDQ) Research Paper. http://www.dama-nl.org/wp- content/uploads/2020/09/DDQ-Dimensions-of-Data-Quality-Research-Paper-version-1.2-d.d.-3-Sept-2020.pdf. DAMA NL Foundation (2020). [Online; accessed 6/10/21].Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

There is value in simplified frameworks that are tailored to a specific use case or technology.

“there is value in simplified frameworks that are tailored to a specific use case or technology.”

• cites [41] 41. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Towards proving the adversarial robustness of deep neural networks. arXiv preprint arXiv:1709.02802 (2017). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to time constraints, the authors did not conduct a systematic review of all possible forward and backward citations.

“Due to time constraints, we did not conduct a systematic review of all possible forward and backward citations.”

Classified as probably not a substantial claim.

The practical techniques mentioned by authors were not reviewed in a systematic manner.

“We did not review these techniques in a systematic manner, as this would merit a separate review of its own.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Traceability makes it possible to interpret and audit the history that precedes the output of ML algorithms.

“Traceability makes it possible to interpret and audit the history that precedes the output of ML algorithms [33]”

• cites [33] 33. Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2020. The dataset nutrition label. Data Protection and Privacy 12, 12 (2020), 1. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

By understanding the requirements that exist, practitioners can be better positioned to select the most meaningful data quality control, assurance and improvement steps for their use case.

“By understanding the requirements that exist, practitioners can be better positioned to select the most meaningful data quality control, assurance and improvement steps for their use case.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These dimensions are subsumed by higher-order characterisations that capture intrinsic, contextual, accessibility, and representational aspects of datasets.

“These dimensions are subsumed by higher-order characterisations that capture the intrinsic, contextual, accessibility, and representational aspects of datasets [74].”

• cites [74] 74. April Yi Wang, Dakuo Wang, Jaimie Drozdal, Xuye Liu, Soya Park, Steve Oney, and Christopher Brooks. 2021. What makes a well-documented notebook? a case study of data scientists' documentation practices in kaggle. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-7. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The general approach to snowballing was informal.

“Our general approach to snowballing was informal.”

Probably not a substantial claim: may not pass S2 (checkable).

Traceability may need to be promoted in the context of ML.

“suggesting that this data quality characteristic may need to be promoted in the context of ML.”

Classified as probably not a substantial claim.

There is no single agreed-upon workflow that is universally applicable to every scenario of ML development.

“Because this is an iterative process that involves numerous decision pathways, there is no single agreed-upon workflow that is universally applicable to every scenario.”

Probably not a substantial claim: may not pass S3 (load-bearing).

They occur at the beginning of the data quality management process defined by ISO 8000-61.

“In other words, they occur at the beginning of the data quality management process defined by the standard ISO 8000-61, as illustrated in Figure 1.”

• cites ISO 8000-61Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This diffusion of responsibility can be addressed by providing clearer indicators about which data quality aspects are and are not out of scope of particular ML roles.

“This diffusion of responsibility can be addressed by providing clearer indicators about which data quality aspects are and are not out of scope of particular ML roles.”

Classified as probably not a substantial claim. May combine two claims.

ML development rarely follows a pre-defined sequence, meaning that data pipelines are difficult to consolidate across different operational contexts.

“Earlier publications and standards acknowledge that ML development rarely follows a pre-defined sequence, meaning that data pipelines are difficult to consolidate across different operational contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Limiting the search only to titles helped to eliminate marginally relevant papers from the results.

“Limiting the search only to titles helped to eliminate marginally relevant papers from the results.”

May combine two claims.

Both of the above challenges are encountered during the initial stage of planning and defining which data qualities to evaluate.

“Both of the above challenges are encountered during the initial stage of planning and defining which data qualities to evaluate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The field of data management already has a good grounding for defining the data quality dimensions that will continue to remain important to ML.

“suggesting that this field already has a good grounding for defining the data quality dimensions that will continue to remain important to ML.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Data users are beginning to assess quality not only in terms of the characteristics of the data (e.g., accuracy), but also the working conditions, skills and aspirations of the individuals who annotated those data.

“In this vein, data users are beginning to assess quality not only in terms of the characteristics of the data (e.g.accuracy), but also the working conditions, skills and aspirations of the individuals who annotated those data [73].”

• cites [73] 73. Jennifer Wortman Vaughan. 2017. Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research. J. Mach. Learn. Res. 18, 1 (2017), 7026-7071.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Understanding the anticipated ML use case and end-user characteristics is needed to design data collection to gather data that adequately reflect their purpose and the environment in which the trained ML will be deployed.

“This understanding is needed to design the data collection process to gather data that adequately reflect their purpose and the environment in which the trained ML will be deployed.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Careful documentation of the use case and underlying assumptions about relevant data attributes can help practitioners and organisations to avoid collecting data signals that may subsequently get discarded.

“Careful documentation of the use case and underlying assumptions about relevant data attributes can help practitioners and organisations to avoid collecting data signals that may subsequently get discarded.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Automated techniques can be used to select the optimal sequence of data preprocessing tasks that maximise the performance of the ML model.

“As mentioned during the pre-processing stage, automated techniques can be used to select the optimal sequence of data preprocessing tasks that maximise the performance of the ML model [11].”

• cites [11] 11. Laure Berti-Equille. 2019. Learn2clean: Optimizing the sequence of tasks for web data preparation. In The World Wide Web Conference. 2580-2586. sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

After data are collected, their contextual integrity must be evaluated and, where necessary, improved through curation of additional data.

“After the data are collected, their contextual integrity must be evaluated and, where necessary, improved through the curation of additional data.”

May combine two claims.

In imbalanced datasets, under-sampling and synthetic data have been found to enhance model performance.

“Here, techniques such as under-sampling and synthetic data have been found to enhance model performance [16].”

• cites [16] 16. Paula Branco, Luís Torgo, and Rita P Ribeiro. 2016. A survey of predictive modeling on imbalanced domains. ACM Computing Surveys (CSUR) 49, 2 (2016), 1-50. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ML can become a tool for detecting smaller subsets of data that would most benefit from enrichment or for using modeling choices to mitigate bias.

“In cases like this, ML in itself can become a tool for detecting smaller subsets of data that would most benefit from enrichment or using modeling choices to mitigate bias [33,34].”

• cites [33] 33. Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2020. The dataset nutrition label. Data Protection and Privacy 12, 12 (2020), 1. sourceNot checked — no full text of the source
• cites [34] 34. Andreas Holzinger. 2018. From machine learning to explainable AI. In 2018 world symposium on digital intelligence for systems and machines (DISA). IEEE, 55-66. sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Distinguishing between different classes of users is necessary to understand the radically different backgrounds and tasks needed to keep ML data quality pipelines running smoothly.

“Distinguishing between these different classes of users is necessary if we are to understand the radically different backgrounds and tasks that are needed to keep ML data quality pipelines running smoothly [57].”

• cites [57] 57. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research 22 (2021). sourceNot checked — no full text of the source

May combine two claims.

Re-sampling approaches can be costly to implement and require the practitioner to know in advance which features are responsible for the undesirable bias.

“One of the downsides of re-sampling approaches is that they can be costly to implement and require the practitioner to know in advance which features are responsible for the undesirable bias [34].”

• cites [34] 34. Andreas Holzinger. 2018. From machine learning to explainable AI. In 2018 world symposium on digital intelligence for systems and machines (DISA). IEEE, 55-66. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Preemptive screening and training of data workers produced better quality data than what would typically be achieved through automated post-processing of noisy data.

“For example, Mitra et al. [49] and Thakkar et al. [70] discussed the importance of preparatory measures in the form of screening and training of data workers, with Mitra et al. [49] finding that this preemptive approach produced better quality data than what would typically be achieved through automated post-processing of noisy data.”

• cites [49] 49. Jorge Merino, Ismael Caballero, Bibiano Rivas, Manuel Serrano, and Mario Piattini. 2016. A data quality in use model for big data. Future Generation Computer Systems 63 (2016), 123-130. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The review did not systematically review the literature on how data quality plans can be implemented through tools for data quality control, assurance and improvement.

“We did not systematically review the literature on how these plans can be implemented through tools for data quality control, assurance and improvement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The ML pipelines illustrated in Figures 4 and 5, along with the sequence of stages in Table 3, use a linear sequence of data quality assurance steps that is unlikely to be structured like this in reality.

“For example the ML pipelines illustrated in Figures 4 and 5 (along with the sequence of stages in Table 3) use a linear sequence of data quality assurance steps that is unlikely to be structured like this in reality.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Runtime verification tools can be used to detect contextual drift.

“Runtime verification tools can be used to detect contextual drift.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Documentation facilitates the handover of information across different stages of ML development.

“Documentation facilitates the handover of information across different stages of ML development.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The review did not do justice to the important trade-offs and negotiations that occur when some parts of data quality may need to be adapted or sacrificed in favour of practical requirements and business goals.

“So we did not do justice to the important trade-offs and negotiations that occur when some parts of data quality may need to be adapted or sacrificed in favour of practical requirements and business goals.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Thresholds for monitoring can be based on the expected distribution of a targeted population for relevant features.

“Chen et al. [19] suggest that the thresholds can be based on the expected distribution of a targeted population for relevant features (e.g. the usage frequency of a phrase, or the number of individuals with a particular skin tone).”

• cites [19] 19. Wo Chang. 2022. ISO/IEC JTC 1/SC 42(AI)/WG 2(Data) Data Quality for Analytics and Machine Learning (ML). https://jtc1info.org/wp-content/uploads/2022/06/01_06_Wo_2022_05_24_ISO-IEC-JTC1-SC42-WG2-Data- Quality-for-Analytics-and-Machine-Learning-Wo-Chang-NIST-final.pdf. [Online; accessed 01/12/2022]. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Visualisations and dashboards can make it easier to inspect the quality of a dataset.

“Visualisations and dashboards can make it easier to inspect the quality of a dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

It is difficult to establish which columns must be inspected and what the required thresholds should be for monitoring.

“However, others note the difficulty in trying to establish which columns must be inspected, and what the required thresholds should be [63].”

• cites [63] 63. Yuji Roh, Geon Heo, and Steven Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering 33, 4 (2019), 1328-1347.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Much of the review's discussion focused on desirable or ideal scenarios rather than what is feasible.

“Moreover, much of our discussion focused on desirable or ideal scenarios rather than what is feasible.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Documenting the environment in which data were collected helps practitioners to assess contextual relevance and to mitigate bias.

“Documenting the environment in which data were collected helps practitioners to assess contextual relevance and to mitigate bias.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Many of the processes described in the review span multiple stakeholders, and stakeholders' ability to self-organise into a robust data quality workflow requires support from higher-level institutional structures.

“Many of the processes described above span across multiple stakeholders, whose ability to selforganise into a robust data quality workflow will require the support of higher-level institutional structures.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is possible that the keywords and sources used during the search, as well as the insights drawn from them, were influenced by the authors' fields of expertise.

“It is possible that the keywords and sources that were used during our search for articles, as well as the insights drawn from them, were influenced by the authors' fields of expertise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review did not elaborate greatly on concepts such as data ethics, data feminism and data justice, which relate to data quality but lie outside the technical focus adopted.

“For example, we did not elaborate greatly on concepts such as data ethics, data feminism and data justice, which relate to data quality but lie outside the technical focus adopted in our review.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Data collection and management software can be used to improve the intrinsic quality of data, for example through runtime verification and alerts.

“Data collection and management software can be used to improve the intrinsic quality of data (e.g. through runtime verification and alerts).”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Clarity about data quality requirements is needed to prevent the diffusion of responsibility and to ensure that every stakeholder is proactive at mitigating data quality issues within their capacity.

“This type of clarity is needed to prevent the diffusion of responsibility and to ensure that every stakeholder is proactive at mitigating data quality issues that are within their capacity.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Documentation of a dataset and its development process can help to anticipate and prevent ethical or legal risks.

“Documentation of the dataset and its development process can help to anticipate and prevent ethical or legal risks.”

Classified as probably not a substantial claim.

The review focused mainly on defining the requirements of ML data as part of the planning stage of the data quality management process illustrated in Figure 1.

“Our review focused mainly on defining the requirements of ML data as part of the “planning” stage of the data quality management process illustrated in Figure 1.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multi-dataset-multi-model and agile data iteration scenarios are more common than the waterfall-ish view used to report the review's findings.

“Specifically, multi-dataset-multi-model and agile data iteration scenarios are more common than the waterfall-ish view used to report our findings.”