Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Understanding and Visualizing Data Iteration in Machine Learning

Open access
Hohman, Fred, Wongsuphasawat, Kanit, Kery, Mary Beth, Patel, Kayur · 2020 · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Abstract

Successful machine learning (ML) applications require iterations on both modeling and the underlying data. While prior visualization tools for ML primarily focus on modeling, our interviews with 23 ML practitioners reveal that they improve model performance frequently by iterating on their data (e.g., collecting new data, adding labels) rather than their models. We also identify common types of data iterations and associated analysis tasks and challenges. To help attribute data iterations to model performance, we design a collection of interactive visualizations and integrate them into a prototype, Chameleon, that lets users compare data features, training/testing splits, and performance across data versions. We present two case studies where developers apply \system to their own evolving datasets on production ML projects. Our interface helps them verify data collection efforts, find failure cases stretching across data versions, capture data processing changes that impacted performance, and identify opportunities for future data iterations.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3313831.3376177

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An overview of a typical machine learning process, which involves both model iteration (e.g., changing model architectures or hyperparameters) and data iteration (e.g., collecting new data to improve new model performance). This paper focuses on data iteration as its tooling is underexplored compared to model iteration.
Figure 1 An overview of a typical machine learning process, which involves both model iteration (e.g., changing model architectures or hyperparameters) and data iteration (e.g., collecting new data to improve new model performance). This paper focuses on data iteration as its tooling is underexplored compared to model iteration.CC-BY · From the PDF
Figure 2 The CHAMELEON interface integrates multiple coordinated views to help practitioners explore their evolving datasets. The Data Version Timeline (A) lists data versions across the top of the interface and allows users to select a primary and a secondary version to visualize below. The Sidebar (B) shows version summaries and multiples views that visualize changing instance predictions. Practitioners can use the sibebar views to filter data in the Feature View (C), which visualizes each feature of a dataset as a histogram with both selected data versions, faceted by performance and the train/testing split. Dataset and feature names are redacted for anonymity.
Figure 2 The CHAMELEON interface integrates multiple coordinated views to help practitioners explore their evolving datasets. The Data Version Timeline (A) lists data versions across the top of the interface and allows users to select a primary and a secondary version to visualize below. The Sidebar (B) shows version summaries and multiples views that visualize changing instance predictions. Practitioners can use the sibebar views to filter data in the Feature View (C), which visualizes each feature of a dataset as a histogram with both selected data versions, faceted by performance and the train/testing split. Dataset and feature names are redacted for anonymity.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
A. The Aggregated Embedding allows users to identify clusters of similar instances using dimensionality reduction.
A. The Aggregated Embedding allows users to identify clusters of similar instances using dimensionality reduction.CC-BY · From the PDF
B. The Prediction Change Matrix reveals how instance predictions change between a pair of data versions.
B. The Prediction Change Matrix reveals how instance predictions change between a pair of data versions.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 5 An instance's sensitivity denotes the number of its prediction changes throughout data versions. Instance 6510 has a sensitivity of 1, changing from incorrectly to correctly predicted (model improvement). Instance 2457 also has a sensitivity of 1, but flips from correctly to incorrectly predicted (model regression). Lastly, instance 583 has a sensitivity of 4, indicating that the instance is sensitive to model changes.
Figure 5 An instance's sensitivity denotes the number of its prediction changes throughout data versions. Instance 6510 has a sensitivity of 1, changing from incorrectly to correctly predicted (model improvement). Instance 2457 also has a sensitivity of 1, but flips from correctly to incorrectly predicted (model regression). Lastly, instance 583 has a sensitivity of 4, indicating that the instance is sensitive to model changes.CC-BY · From the PDF
Figure 6 A feature's long-tailed, multi-modal distribution shape solidifies over collection time, from 1,442 (ver. 1.1 ) to 64,205 (ver. 1.18 ) instances.
Figure 6 A feature's long-tailed, multi-modal distribution shape solidifies over collection time, from 1,442 (ver. 1.1 ) to 64,205 (ver. 1.18 ) instances.CC-BY · From the PDF
Figure 7 Filtering across features helps practitioners quickly find interesting data subsets to compare against the global distributions, e.g., this subset found using the Aggregated Embedding that performs well (91.2% accuracy) and has feature values strictly greater than 75.
Figure 7 Filtering across features helps practitioners quickly find interesting data subsets to compare against the global distributions, e.g., this subset found using the Aggregated Embedding that performs well (91.2% accuracy) and has feature values strictly greater than 75.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Research works printed but not linked: 1

  1. Matthias Feurer Aaron Klein Katharina Eggensperger Jost Springenberg Manuel Blum and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems. 2962--2970.

Cited by

See all 125

Metadata

Authors
Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K.
Affiliations
  • Georgia Institute of Technology (US)
  • Apple (United States) (US)
  • Carnegie Mellon University (US)
Venue
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2020-04-21
Pages
1-13
OpenAlex
W3032086959
Cited by
110
References
48
Citations by year
2026: 5 · 2025: 16 · 2024: 24 · 2023: 19 · 2022: 19 · 2021: 22 · 2020: 5
FWCI
5.01
Type
conference paper
Language
en
Open access
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
interactive visualization, data versioning, training-test split, data collection, data annotation, failure case analysis, machine learning in production

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Understanding and Visualizing Data Iteration in Machine Learning. https://app.cochat.ai/papers/10.1145/3313831.3376177
The paper
Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K. (2020). Understanding and Visualizing Data Iteration in Machine Learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376177

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.