The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Understanding and Visualizing Data Iteration in Machine Learning
Open accessAbstract
Successful machine learning (ML) applications require iterations on both modeling and the underlying data. While prior visualization tools for ML primarily focus on modeling, our interviews with 23 ML practitioners reveal that they improve model performance frequently by iterating on their data (e.g., collecting new data, adding labels) rather than their models. We also identify common types of data iterations and associated analysis tasks and challenges. To help attribute data iterations to model performance, we design a collection of interactive visualizations and integrate them into a prototype, Chameleon, that lets users compare data features, training/testing splits, and performance across data versions. We present two case studies where developers apply \system to their own evolving datasets on production ML projects. Our interface helps them verify data collection efforts, find failure cases stretching across data versions, capture data processing changes that impacted performance, and identify opportunities for future data iterations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708 · cited 1×
“During exploratory data analysis [43, 14], users may be unfamiliar with their resources (e.g., data), uncertain of their goals, and unsure how to reach them.”
BackgroundTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101 · cited 2×
“For reduction and visualization, the Aggregated Embedding uses UMAP [26]: a non-linear dimensionality reduction that better preserves global data structure compared to other techniques like t-SNE [25].”
MethodLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Software Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 1×
“A recent study of ML industry professionals found that “discovering, managing, and versioning the data needed for machine learning applications is much more complex and difficult than other types of software engi-neering” [3].”
BackgroundPower to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · cited 3×
“Successful machine learning (ML) applications require an iterative process to create models that deliver desired performance and user experience [4, 38].”
BackgroundA Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective
Roh, Y., Heo, G., & Whang, S. E. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2946162 · Cited by 968 · cited 4×
“This is corroborated in prior data collection work [35].”
BackgroundResultInfluentialFaceted metadata for image search and browsing
Yee, K.-P., Swearingen, K., Li, K., & Hearst, M. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642681 · Cited by 898 · cited 1×
“Faceted browsing [47] lets users use metadata to filter subsets of data that share desired properties.”
BackgroundVega-Lite: A Grammar of Interactive Graphics
Satyanarayan, A., Moritz, D., Wongsuphasawat, K., & Heer, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2599030 · Cited by 883 · cited 1×
“Visual-izations were constructed and rendered using Vega-Lite [37].”
MethodSelf-citation
Research works printed but not linked: 1
- Matthias Feurer Aaron Klein Katharina Eggensperger Jost Springenberg Manuel Blum and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems. 2962--2970.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
“identified common types of data iterations and created a tool to visualise them [52].”
BackgroundMethodTrustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
“Some researchers [89, 179, 182] proposed visualization techniques to better understand the data before using it.”
BackgroundTransfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning
Prottasha, N. J., Sami, A. A., Kowsher, M., Murad, S. A., Bairagi, A. K., Masud, M., & Baz, M. (2022) · Sensors · doi:10.3390/s22114157 · Cited by 248
“Data preparation is essential in machine learning-based classification, as the model’s accuracy is heavily dependent on the quality of the input data [33].”
BackgroundTechnology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209
“For instance, a more iterative process for Bayesian ML [60] and even more specifically for probabilistic programming [39], a data mining process defined in 2000 that remains widely used [61], others for describing data iterations [62], and human-computer interaction cycles [63].”
Metadata
- Authors
- Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K.
- Affiliations
- Georgia Institute of Technology (US)
- Apple (United States) (US)
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3032086959
- Semantic Scholar
- d6b9a5a2af3c2f9044c505855b27310132b7f3d4
- Cited by
- 110
- References
- 48
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 24 · 2023: 19 · 2022: 19 · 2021: 22 · 2020: 5
- FWCI
- 5.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- interactive visualization, data versioning, training-test split, data collection, data annotation, failure case analysis, machine learning in production
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding and Visualizing Data Iteration in Machine Learning. https://app.cochat.ai/papers/10.1145/3313831.3376177
- The paper
- Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K. (2020). Understanding and Visualizing Data Iteration in Machine Learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376177
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.