Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

Most checklists currently in use are not developed for ML-based science in general but rather for specific scientific or research communities.

“Further, most checklists currently in use are not developed for ML-based science in general but rather for specific scientific or research communities.4,38”

Standards of this kind crystallise an ever growing corpus of academic literature that has explored ML data quality challenges and ways to mitigate them.

“Standards of this kind crystallise an ever growing corpus of academic literature that has explored ML data quality challenges and ways to mitigate them [25,57,64,70].”

A singular focus on immediate data users can flatten the variety of uses and data quality requirements encountered at various stages of the ML development pipeline over the longer term.

“However, this singular focus can flatten the variety of uses and data quality requirements that are encountered at various stages of the ML development pipeline over the longer-term [57,70].”

Distinguishing between different classes of users is necessary to understand the radically different backgrounds and tasks needed to keep ML data quality pipelines running smoothly.

“Distinguishing between these different classes of users is necessary if we are to understand the radically different backgrounds and tasks that are needed to keep ML data quality pipelines running smoothly [57].”

Data quality must be defined using stage-specific approaches that are sensitive to where in the ML lifecycle the data are encountered and who encounters them.

“Instead, data quality must be defined using stage-specific approaches that are sensitive to where in the ML lifecycle the data are encountered and who encounters them [25,57].”

Managing data quality in industrial use cases may require new infrastructure that ingests data and converts them into a form that is more compatible with the ML trainer.

“As a result of the above, managing data quality in industrial use cases may require new infrastructure that ingests data and converts them into a form that is more compatible with the ML trainer [57].”

Traditional data quality approaches designed for relational and static datasets may not be sufficient when dealing with large-scale decentralised ML pipelines.

“Several authors have therefore noted that traditional data quality approaches designed for relational and static datasets may not be sufficient when dealing with the kinds of large-scale decentralised ML pipelines that are increasingly being used for operational and organisation-wide decision making [25,57].”

Earlier trends of opportunistic data collection and post hoc justifications of large datasets are gradually moving towards more deliberative data collection methods, sampling techniques and minimal data architectures to deliver better performance without reducing model accuracy.

“Earlier trends of opportunistic data collection and post hoc justifications of large datasets are gradually moving towards a requirement for more deliberative data collection methods [35,36], sampling techniques [16] and minimal data architectures [57] to deliver better performance without reducing model accuracy.”

Training-serving skew can be detected in pre-defined variables using analyses proposed by Polyzotis et al. and Schelter et al.

“Polyzotis et al. [57] and Schelter et al. [65] propose analyses that can be used to detect training-serving skew in pre-defined variables.”