The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Open accessAbstract
Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Cabrera et al.'s Zeno platform [69] provides an interactive framework to probe AI systems for systematic failures and biases by examining performance on specific data subgroups, directly addressing the need for rigorous stress-testing tools.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [69]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Cabrera et al. [69] introduced the concept of 'behavior descriptions' as a form of explanation and evaluated how these influence user understanding.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [69]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-00a · Cited by 11,032
Computational Radiomics System to Decode the Radiographic Phenotype
(2017) · doi:10.1158/0008-5472.can-17-0339 · Cited by 6,593 · cited 1×
“Since the participant was not a domain expert, they consulted with medical imaging researchers that recommended a Python library, pyradiomics [55], to extract physiologically relevant characteristics from medical images.”
Background(2018) · Cited by 3,510
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Machine learning (ML) systems deployed in the real world can encode problems such as societal biases [3] and safety concerns [41].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Kolb, T. M., Lichy, J., & Newhouse, J. H. (2002) · Radiology · doi:10.1148/radiol.2251011667 · Cited by 1,718 · cited 1×
“performed signifcantly worse for images with higher tissue density, a phenomenon that also occurs with human radiologists [30].”
BackgroundKvasir-SEG: A Segmented Polyp Dataset
Jha, D., Smedsrud, P. H., Riegler, M. A., Halvorsen, P., de Lange, T., Johansen, D., & Johansen, H. D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-37734-2_37 · Cited by 1,661
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects
Ni, J., Li, J., & McAuley, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1018 · Cited by 1,200 · cited 1×
“zeno currently has six implemented views, shown here with the following datasets: image classifcation (CIFAR-10 [31]), audio transcription (Free Spoken Digit Dataset [27]), image segmentation (Kvasir-SEG [28]), text classifcation (Amazon reviews [42]), timeseries classifcation (MotionSense [34]), and object detection…”
BackgroundA unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029 · cited 1×
“A common phenomenon for models deployed in the real world is domain shift, where the real-world data distribution changes over time and model performance degrades [37].”
BackgroundBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Vega-Lite: A Grammar of Interactive Graphics
Satyanarayan, A., Moritz, D., Wongsuphasawat, K., & Heer, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2599030 · Cited by 883 · cited 1×
“The Python package includes the compiled frontend which is written in Svelte and uses Vega-Lite [52] for visualizations and Arquero [24] for data manipulation.”
MethodHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
National Transportation Safety Board
(2017) · The SAGE International Encyclopedia of Travel and Tourism · doi:10.4135/9781483368924.n320 · Cited by 725
Metamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
The What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viegas, F., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 505 · cited 1×
“The What-If tool [57] is another interactive framework that focuses on using counterfactuals to understand model behavior and fx fairness concerns.”
MethodFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411 · cited 1×
“Practitioners and researchers continue to discover signifcant limitations and failures in state-of-the-art models, from systematic misclassifcation of certain medical images [44] to racial biases in pedestrian detection models [58].”
BackgroundPredictive Inequity in Object DetectionPreprint
Wilson, B., Hoffman, J., & Morgenstern, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11097 · Cited by 369
Sagawa, S., Koh, P. W., Hashimoto, T., & Liang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08731 · Cited by 331 · cited 1×
“For example, there are promising results in using data slices to improve model performance, such as slice-based learning [14] and group distributionally robust optimization (GDRO) [33, 51].”
BackgroundJust Train Twice: Improving Group Robustness without Training Group InformationPreprint
Liu, E. Z., Haghgoo, B., Chen, A., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09044 · Cited by 322 · cited 1×
“For example, there are promising results in using data slices to improve model performance, such as slice-based learning [14] and group distributionally robust optimization (GDRO) [33, 51].”
BackgroundMobile sensor data anonymization
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2019) · Proceedings of the International Conference on Internet of Things Design and Implementation · doi:10.1145/3302505.3310068 · Cited by 298 · cited 1×
“zeno currently has six implemented views, shown here with the following datasets: image classification (CIFAR-10 [31]), audio transcription (Free Spoken Digit Dataset [27]), image segmentation (Kvasir-SEG [28]), text classification (Amazon reviews [42]), timeseries classification (MotionSense [34]), and object detecti…”
BackgroundRay: A Distributed Framework for Emerging AI Applications
Moritz, P., Nishihara, R., Wang, S. F., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05889 · Cited by 247 · cited 1×
“A potential update would be to support processing in distributed computing clusters using a library such as Ray [39].”
MethodDomino: Discovering Systematic Errors with Cross-Modal EmbeddingsPreprint
Eyuboglu, S., Varma, M., Saab, K. K., Delbrouck, J.-B., Lee‐Messer, C., Dunnmon, J., Zou, J., & Ré, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.14960 · Cited by 207 · cited 1×
“Often, there is not enough metadata to defne slices with high error, so another family of methods uses model embeddings and clustering to fnd groups with high error [18, 19].”
MethodRobustness Gym: Unifying the NLP Evaluation Landscape
Goel, K., Rajani, N. F., Vig, J., Taschdjian, Z., Bansal, M., & Ré, C. (2021) · doi:10.18653/v1/2021.naacl-demos.6 · Cited by 197
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., & Chau, D. H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.51 · Cited by 171
Errudite: Scalable, Reproducible, and Testable Error Analysis
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1073 · Cited by 141 · cited 1×
“Errudite [59] is a similar system for NLP models with which users can create and test subgroups using structured queries.”
MethodTowards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126 · cited 1×
“Lastly, there are approaches that use end-user reports or crowd feedback to discover model failures or interesting behaviors [2, 8, 43]. zeno complements discovery methods by allowing users to formalize, validate, and track hypotheses of systemic errors over time.”
BackgroundAdaptive Testing and Debugging of NLP Models
Ribeiro, M. T., & Lundberg, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.230 · Cited by 116 · cited 1×
“automated error discovery methods such as fnding clusters with high error or using foundational models [5, 48] to generate new instances, but still primarily rely on human-generated feedback.”
MethodThe Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models
d'Eon, G., d’Eon, J., Wright, J. R., & Leyton‐Brown, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533240 · Cited by 111 · cited 1×
“Often, there is not enough metadata to defne slices with high error, so another family of methods uses model embeddings and clustering to fnd groups with high error [18, 19].”
MethodNeo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels
Görtler, J., Hohman, F., Moritz, D., Wongsuphasawat, K., Ren, D., Nair, R. S., Kirchner, M., & Patel, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501823 · Cited by 104 · cited 1×
“…such as DendroMap [4], Facets [46], or AnchorViz [12], could improve users’ ability to find patterns and new behaviors in their data. zeno can also adapt existing visualizations of ML performance, such as ML Cube [29], Neo [23], or ConfusionFlow [25], to better visualize model behaviors.”
BackgroundInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101 · cited 2×
“Extensions to these reporting methods, namely Interactive Model Cards [16], have aimed to improve their usability by making them more expressive and interactive.”
MethodExploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · cited 1×
“Furthermore, many behavioral evaluation tools, such as fairness toolkits, often do not support the types of models, data, and behaviors that practitioners work with in the real world [17].”
BackgroundSelf-citationVisual exploration of machine learning results using data cube analysis
Kahng, M., Fang, D., & Chau, D. H. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939503 · Cited by 88 · cited 2×
“Instance views that encode semantic similarity, such as DendroMap [4], Facets [46], or AnchorViz [12], could improve users’ ability to find patterns and new behaviors in their data. zeno can also adapt existing visualizations of ML performance, such as ML Cube [29], Neo [23], or ConfusionFlow [25], to better visualize…”
BackgroundSlice Finder: Automated Data Slicing for Model Validation
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019) · 2019 IEEE 35th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00139 · Cited by 83
Discovering and Validating AI Errors With Crowdsourced Failure Reports
Cabrera, Á. A., Druck, A. J., Hong, J., & Perer, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479569 · Cited by 83 · cited 3×
“A common collaboration challenge is making sense of failure reports [8] from end-users.”
BackgroundSelf-citationTowards A Process Model for Co-Creating AI Experiences
Subramonyam, H., Seifert, C. M., & Adar, E. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462012 · Cited by 77 · cited 2×
“have as they iterate on and deploy their ML systems [54].”
BackgroundRequirements Engineering for Machine Learning: A Review and Reflection
Pei, Z., Liu, L., Wang, C., & Wang, J. (2022) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew56159.2022.00039 · Cited by 73 · cited 1×
“Inspired by requirements engineering in software engineering, behavioral evaluation focuses on defning and testing the capabilities of an ML system, its expected behavior on a specifcation of requirements [45, 60].”
MethodSymphony: Composing Interactive Interfaces for Machine Learning
Bäuerle, A., Cabrera, Á. A., Hohman, F., Maher, M., Koski, D., Suau, X., Barik, T., & Moritz, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502102 · Cited by 69 · cited 2×
“framework, Symphony [7], introduces modular data and model analysis components that can be used in both computational notebooks”
MethodSelf-citationConfusionFlow: A Model-Agnostic Visualization for Temporal Analysis of Classifier Confusion
Hinterreiter, A., Ruch, P., Stitz, H., Ennemoser, M., Bernard, J., Strobelt, H., & Streit, M. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3012063 · Cited by 67 · cited 2×
“Instance views that encode semantic similarity, such as DendroMap [4], Facets [46], or AnchorViz [12], could improve users’ ability to find patterns and new behaviors in their data. zeno can also adapt existing visualizations of ML performance, such as ML Cube [29], Neo [23], or ConfusionFlow [25], to better visualize…”
BackgroundWhat Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 62 · cited 3×
“To detect andmitigate these important issues, theML community uses more fine-grained evaluation approaches, often termed behavioral evaluation [10, 47].”
BackgroundMethodInfluentialSelf-citationMarcelle: Composing Interactive Machine Learning Workflows and Interfaces
Françoise, J., Caramiaux, B., & Sanchez, T. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474734 · Cited by 62 · cited 1×
“Marcelle [20] similarly uses modular components that allow users to modify an ML pipeline without writing code.”
MethodFalcon: Balancing Interactive Latency and Resolution Sensitivity for Scalable Linked Visualizations
Moritz, D., Howe, B., & Heer, J. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300924 · Cited by 60 · cited 2×
“zeno could implement an optimization strategy like Falcon [38] to support live cross-fltering on large datasets.”
Beat the machine: Challenging workers to find the unknown unknowns
Attenberg, J., Ipeirotis, P. G., & Provost, F. J. (2011) · Cited by 57
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Chen, N.-C., Suh, J., Verwey, J., Ramos, G., Drucker, S. M., & Simard, P. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172950 · Cited by 49 · cited 2×
“Instance views that encode semantic similarity, such as DendroMap [4], Facets [46], or AnchorViz [12], could improve users’ ability to find patterns and new behaviors in their data. zeno can also adapt existing visualizations of ML performance, such as ML Cube [29], Neo [23], or ConfusionFlow [25], to better visualize…”
BackgroundJakobovski/free-spoken-digit-dataset v1.0.4
Jackson, Z., Souza, C., Flaks, J., Pan, Y., Nicolas, H., & Thite, A. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1000741 · Cited by 47
Jakobovski/free-spoken-digit-dataset: v1.0.8
Jackson, Z., de Souza, C. R., Flaks, J., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1342401 · Cited by 45
DendroMap: Visual Exploration of Large-Scale Image Datasets for Machine Learning with Treemaps
Bertucci, D., Hamid, M. M., Anand, Y., Ruangrotsakun, A., Tabatabai, D., Perez, M., & Kahng, M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209425 · Cited by 45 · cited 2×
“Instance views that encode semantic similarity, such as DendroMap [4], Facets [46], or AnchorViz [12], could improve users’ ability to find patterns and new behaviors in their data. zeno can also adapt existing visualizations of ML performance, such as ML Cube [29], Neo [23], or ConfusionFlow [25], to better visualize…”
BackgroundSelf-citationSlice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
Chen, V. S., Wu, S., Weng, Z., Ratner, A., & Ré, C. (2019) · PubMed · doi:10.48550/arxiv.1909.06349 · Cited by 42 · cited 1×
“For example, there are promising results in using data slices to improve model performance, such as slice-based learning [14] and group distributionally robust optimization (GDRO) [33, 51].”
BackgroundHopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40 · cited 2×
“Practitioners end up manually testing hand-picked examples from users and stakeholders, making it challenging to efectively compare models and pick the best version to deploy [26].”
BackgroundNutritional Labels for Data and Models
Stoyanovich, J., & Howe, B. (2019) · IEEE Data(base) Engineering Bulletin · Cited by 37
Did the Model Change? Efficiently Assessing Machine Learning API ShiftsPreprint
Chen, L., Cai, T., Zaharia, M., & Zou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.14203 · Cited by 19 · cited 1×
“Model updates are even more complicated for teams that rely on external AI services, as practitioners do not control when or how services are updated [11].”
BackgroundNahar, N., Zhou, S., Lewis, G. A., & Kästner, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10234 · Cited by 18 · cited 3×
“Models are often developed without practitioners having clear model requirements or a deep understanding of the products or services in which the model will be deployed [40].”
BackgroundFAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · cited 2×
“An example tool for slice-based analysis is FairVis [9], a visual analytics system that allows users to compare subsets of data across metrics to discover intersectional biases.”
MethodSelf-citationDendroMap: Visual Exploration of Large-Scale Image Datasets for Machine Learning with TreemapsPreprint
Bertucci, D., Hamid, M. M., Anand, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.06935 · Cited by 6 · cited 2×
“Instance views that encode semantic similarity, such as DendroMap [4], Facets [46], or AnchorViz [12], could improve users’ ability to find patterns and new behaviors in their data. zeno can also adapt existing visualizations of ML performance, such as ML Cube [29], Neo [23], or ConfusionFlow [25], to better visualize…”
BackgroundSelf-citationCapabilities for Better ML EngineeringPreprint
Yang, C., Brower–Sinning, R., Lewis, G. A., Kästner, C., & Wu, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.06409 · Cited by 5
The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models
d'Eon, G., d'Eon, J., Wright, J. R., & Leyton-Brown, K. (2021) · arXiv · doi:10.48550/arxiv.2107.00758 · cited 1×
“Often, there is not enough metadata to defne slices with high error, so another family of methods uses model embeddings and clustering to fnd groups with high error [18, 19].”
MethodRobustness Gym: Unifying the NLP Evaluation Landscape
Goel, K., Rajani, N., Vig, J., Tan, S., Wu, J., Zheng, S., Xiong, C., Bansal, M., & Ré, C. (2021) · arXiv · doi:10.48550/arxiv.2101.04840 · cited 1×
“For instance, Robustness Gym [22] is a framework for NLP models that supports multiple types of evaluation, including adversarial attacks and robustness checks.”
Method
Research works printed but not linked: 2
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y., & Kim, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642216 · Cited by 204
“For example, Zeno [11], the What-If Tool [65], and Errudite [67] help practitioners to identify slices or subsets of data that may reveal distinct model failures.”
EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y.-H., & Kim, J. (2023) · arXiv · doi:10.48550/arxiv.2309.13633 · Cited by 82
“For example, Zeno [11], the What-If Tool [65], and Errudite [67] help practitioners to identify slices or subsets of data that may reveal distinct model failures.”
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025) · Informatics · doi:10.3390/informatics12040119 · Cited by 39
“Related concepts such as co-design and participatory frameworks for AI development [56], new behavioral evaluation tools (e.g., Zeno [69]), and emerging ethical and governance principles for XAI [11] also enrich this evolving landscape.”
Interactive Debugging and Steering of Multi-Agent AI Systems
Epperson, W., Bansal, G., Dibia, V. C., Fourney, A., Gerrits, J., Zhu, E. (., & Amershi, S. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713581 · Cited by 35
TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students
Jin, H., Yoo, M., Park, J., Lee, Y., Wang, X., & Kim, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714054 · Cited by 75
“Thirty science teachers designed PCAs in a between-subjects study, and using TeachTune resulted in a lower task load and higher student profile coverage over a baseline.”
Metadata
- Authors
- Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A.
- Affiliations
- Carnegie Mellon University (US)
- Oregon State University (US)
- ORCID
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-14
- OpenAlex
- W4320342906
- Cited by
- 51
- References
- 60
- Citations by year
- 2026: 8 · 2025: 23 · 2024: 15 · 2023: 5
- FWCI
- 9.03
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (IIS-2040942)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- systematic failures, AI safety, algorithmic bias, interactive visualization, AI testing, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.