Discovering and Validating AI Errors With Crowdsourced Failure Reports
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
AI systems can fail to learn important behaviors, leading to real-world issues like safety concerns and biases. Discovering these systematic failures often requires significant developer attention, from hypothesizing potential edge cases to collecting evidence and validating patterns. To scale and streamline this process, we introduce crowdsourced failure reports, end-user descriptions of how or why a model failed, and show how developers can use them to detect AI errors. We also design and implement Deblinder, a visual analytics system for synthesizing failure reports that developers can use to discover and validate systematic failures. In semi-structured interviews and think-aloud studies with 10 AI practitioners, we explore the affordances of the Deblinder system and the applicability of failure reports in real-world settings. Lastly, we show how collecting additional data from the groups identified by developers can improve model performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
UMAP: Uniform Manifold Approximation and Projection
McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018) · Journal of Open Source Software · doi:10.21105/joss.00861 · Cited by 9,826
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.00982 · Cited by 3,653
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 3×
“Unfortunately, discovering what systematic errors AI systems make presents developers with various challenges [2, 25, 31].”
BackgroundData scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
Research works printed but not linked: 56
- 10.1109/TVCG.2019.2934262 · DOI · Paper page
- 10.1109/ICSE-SEIP.2019.00042 · DOI · Paper page
- 10.1145/2531602.2531653 · DOI · Paper page
- Josh M Attenberg, Pagagiotis G Ipeirotis, and Foster Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.
- Gagan Bansal and Daniel S. Weld. 2018. A coverage-based utility model for identifying unknown unknowns. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 (2018), 1463--1470.
- 10.1016/b978--155860915-0/50016-0 · doi:10.1016/b978--155860915-0/50016-0
- 10.1145/1453101.1453146 · DOI · Paper page
- Nicolas Bettenburg, Rahul Premraj, Thomas Zimmermann, and Sunghun Kim. [n.d.]. Duplicate bug reports considered harmful? really?. In 2008 IEEE International Conference on Software Maintenance. IEEE, 337--345.
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. 77--91.
- 10.1109/VAST47406.2019.8986948 · DOI · Paper page
- Ángel Alexander Cabrera Fred Hohman Jason Lin and Duen Horng Chau. 2018. Interactive Classification for Deep Learning Interpretation. (2018) 1--5. http://arxiv.org/abs/1806.05660 · arXiv
- 10.1145/2858036.2858411 · DOI
- 10.1145/1978942.1978967 · DOI · Paper page
- 10.1145/3172944.3172950 · DOI · Paper page
- Vincent S. Chen, Sen Wu, Zhenzhen Weng, Alexander Ratner, and Christopher Ré. 2019. Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices. NeurIPS (2019). http://arxiv.org/abs/1909.06349 · arXiv
- 10.1145/2675133.2675214 · DOI · Paper page
- 10.1109/TVCG.2017.2746018 · DOI
- 10.1145/2207676.2207711 · DOI · Paper page
- 10.1145/3134680 · DOI
- 10.1109/TVCG.2012.324 · DOI
- Nitesh Goyal. 2015. Designing for Collaborative Sensemaking: Using Expert & Non-Expert Crowd. (2015). http://arxiv.org/abs/1511.06053 · arXiv
- 10.14778/3421424.3421429 · DOI
- 10.1109/TR.2018.2873427 · DOI
- 10.1109/ACCESS.2020.2985222 · DOI
- 10.1145/2939502.2939503 · DOI · Paper page
- 10.1145/2531602.2531644 · DOI
- 10.1145/2470654.2481415 · DOI · Paper page
- 10.1109/PACIFICVIS.2015.7156366 · DOI · Paper page
- 10.1007/s11263-020-01316-z · DOI · Paper page
- 10.1609/aaai.v31i1.10821 · DOI · Paper page
- 10.1145/3366423.3380306 · DOI · Paper page
- 10.1109/ICCV.2015.425 · DOI · Paper page
- 10.1145/3415168 · DOI
- 10.3115/v1/D14-1162 · DOI · Paper page
- 10.1037/0033--295X.106.4.643 · doi:10.1037/0033--295x.106.4.643
- 10.1007/s13398-014-0173--7.2 · doi:10.1007/s13398-014-0173--7.2
- 10.1007/s11263-016-0965--7 · doi:10.1007/s11263-016-0965--7
- Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings (2016), 1--16.
- 10.1038/s41586-019--1138-y · doi:10.1038/s41586-019--1138-y
- 10.1609/aaai.v33i01.33016137 · DOI
- 10.1109/TSE.2013.2297712 · DOI · Paper page
- Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data Programming: Creating Large Training Sets, Quickly. In Proceedings of the 30th International Conference on Neural Information Processing Systems (Barcelona, Spain) (NIPS'16). Curran Associates Inc., Red Hook, NY, USA, 3574--3582.
- 10.1109/TVCG.2016.2598828 · DOI · Paper page
- 10.1002/9780470689646.ch1 · DOI · Paper page
- 10.1109/MSR.2010.5463280 · doi:10.1109/msr.2010.5463280
- 10.1145/1357054.1357247 · DOI · Paper page
- Tom Simonite. 2018. When it comes to gorillas, google photos remains blind. Wired, January, Vol. 13 (2018).
- 10.1109/ASE.2011.6100061 · DOI · Paper page
- 10.1145/1806799.1806811 · DOI · Paper page
- 10.2139/ssrn.3063289 · DOI · Paper page
- Daisuke Wakabayashi. 2018. Self-driving Uber car kills pedestrian in Arizona, where robots roam. The New York Times, Vol. 3 (2018), 19.
- Karl E Weick. 1995. Sensemaking in Organizations (Foundations for Organizational Science). Star (1995).
- 10.18653/v1/P19-1073 · DOI · Paper page
- 10.1109/ICSME.2014.66 · doi:10.1109/icsme.2014.66
- 10.1007/s11432-014--5241--2 · doi:10.1007/s11432-014--5241--2
- 10.1109/TSE.2010.63 · DOI · Paper page
Cited by
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
“, [6, 13, 57, 60, 65, 88]), which automatically surface cases that are more likely to be mislabeled and/or misclassified.”
BackgroundDeng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
DeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98
Improving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87
“More recently, crowd-based systems such as Pandora [44], Deblinder [9], and P-BTM [38] have shown how crowdsourcing can augment behavioral analysis tools by surfacing previously unknown model failures.”
BackgroundSelf-citationEvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y.-H., & Kim, J. (2023) · arXiv · doi:10.48550/arxiv.2309.13633 · Cited by 82
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · arXiv · doi:10.48550/arxiv.2304.09991 · Cited by 79
Chung, J., Kamar, E., & Amershi, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.34 · Cited by 72
“…humans extracting features (Branson et al., 2010; Cheng and Bern-stein, 2015), interactively changing the error patterns (Kapoor et al., 2010; Talbot et al., 2009), or interactively testing models (Wu et al., 2019; Yuan et al., 2022; Ribeiro et al., 2020; Cabrera et al., 2021; Suh et al., 2019).”
BackgroundZeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581268 · Cited by 51
“A common collaboration challenge is making sense of failure reports [8] from end-users.”
BackgroundSelf-citationEvaluating the impact of human explanation strategies on human-AI visual decision-making
Morrison, K., Shin, D., Holstein, K., & Perer, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579481 · Cited by 37
Self-citationTurri, V., Morrison, K., Robinson, K.-M., Abidi, C., Perer, A., Forlizzi, J., & Dzombak, R. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658985 · Cited by 10
Self-citationHuman-Centered Evaluation and Auditing of Language Models
Liu, Y. L., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Liao, Q. V., Xu, W., Novikova, J., & Xiao, Z. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706729 · Cited by 7
“…the perspectives and lived experiences of diverse stakeholders in AI evaluation and auditing, there’s growing interest among both HCI and AI researchers in directly involving these stakeholders in the evaluation and auditing of LLMs (as well as other AI systems) [3, 13, 23, 26, 41, 42, 45].”
Metadata
- Authors
- Cabrera, Á. A., Druck, A. J., Hong, J. I., & Perer, A.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3204393347
- Cited by
- 60
- References
- 12
- FWCI
- 16.50
- Type
- journal article
- Language
- en
- Open access
- bronze
- Funders
- National Science Foundation (IIS-2040942)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.