Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Open accessAbstract
As machine learning systems move from computer-science laboratories into the open world, their accountability becomes a high priority problem. Accountability requires deep understanding of system behavior and its failures. Current evaluation methods such as single-score error metrics and confusion matrices provide aggregate views of system performance that hide important shortcomings. Understanding details about failures is important for identifying pathways for refinement, communicating the reliability of systems in different settings, and for specifying appropriate human oversight and engagement. Characterization of failures and shortcomings is particularly complex for systems composed of multiple machine learned components. For such systems, existing evaluation methods have limited expressiveness in describing and explaining the relationship among input content, the internal states of system components, and final output quality. We present Pandora, a set of hybrid human-machine methods and tools for describing and explaining system failures. Pandora leverages both human and system-generated observations to summarize conditions of system malfunction with respect to the input content and system architecture. We share results of a case study with a machine learning pipeline for image captioning that show how detailed performance views can be beneficial for analysis and debugging.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
(2005) · Cited by 4,152 · see citing works
(2018) · Cited by 3,510 · see citing works
A maximum entropy approach to natural language processing
Berger, A. L., Pietra, V. J. D., & Pietra, S. A. D. (1996) · doi:10.5555/234285.234289 · Cited by 3,083 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Support vector machine in machine condition monitoring and fault diagnosis
Widodo, A., & Yang, B.-S. (2007) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2006.12.007 · Cited by 1,650 · see citing works
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · see citing works
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · see citing works
(2015) · neural information processing systems · Cited by 912 · see citing works
How to Explain Individual Classification DecisionsPreprint
Baehrens, D., Schroeter, T. S., Stefan Harmeling, B., Kawanabe, M., Hansen, K., & Mueller, K.-R. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0912.1128 · Cited by 743 · see citing works
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
Model-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699 · see citing works
Research Priorities for Robust and Beneficial Artificial Intelligence
Russell, S., Dewey, D., & Tegmark, M. (2015) · AI Magazine · doi:10.1609/aimag.v36i4.2577 · Cited by 637 · see citing works
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · see citing works
Explainable Artificial Intelligence (XAI)
Hossain, M. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6371435 · Cited by 544 · see citing works
Combining human and machine intelligence in large-scale crowdsourcing
Kamar, E. S., Hacker, S., & Horvitz, E. (2012) · doi:10.5555/2343576.2343643 · Cited by 323 · see citing works
Self-citationInteractive machine learning: letting users build classifiers
WARE, M., Frank, E., Holmes, G., HALL, M., & Witten, I. H. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2001.0499 · Cited by 207 · see citing works
Directions in hybrid intelligence: complementing AI systems with human intelligence
Kamar, E. S. (2016) · International Joint Conference on Artificial Intelligence · Cited by 154 · see citing works
Self-citationEU regulations on algorithmic decision-making and a "right to explanation".Preprint
Goodman, B., & Flaxman, S. (2016) · arXiv (Cornell University) · Cited by 134 · see citing works
A Bayesian Approach to Tackling Hard Computational ProblemsPreprint
Horvitz, E., Ruan, Y., Gomes, C. P., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.2279 · Cited by 116 · see citing works
Self-citationAn Analysis of ISO 26262: Using Machine Learning Safely in Automotive SoftwarePreprint
Salay, R., Queiroz, R., & Czarnecki, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02435 · Cited by 99 · see citing works
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
Nushi, B., Kamar, E. S., Horvitz, E., & Kossmann, D. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.10633 · Cited by 65 · see citing works
Self-citationDecision-Theoretic Troubleshooting: A Framework for Repair and Experiment
Breese, J. S., & Heckerman, D. E. (1999) · Springer eBooks · doi:10.1007/978-1-4615-5089-1_15 · Cited by 61 · see citing works
Beat the machine: Challenging workers to find the unknown unknowns
Attenberg, J., Ipeirotis, P. G., & Provost, F. J. (2011) · Cited by 57 · see citing works
Towards Transparent Systems: Semantic Characterization of Failure Modes
Bansal, A., Farhadi, A., & Parikh, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_24 · Cited by 41 · see citing works
Interpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · see citing works
Self-citationMini-crowdsourcing end-user assessment of intelligent assistants: A cost-benefit study
Shinsel, A., Kulesza, T., Burnett, M., Curran, W., Groce, A., Stumpf, S., & Wong, W. (2011) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2011.6070377 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- [10]10. Buolamwini, J., and Gebru, T. 2018. Gender shades: In- tersectional accuracy disparities in commercial gender clas- sification. In Conference on Fairness, Accountability and Transparency, 77-91.
- [11]11. Caruana, R.; Lou, Y.; Gehrke, J.; Koch, P.; Sturm, M.; and Elhadad, N. 2015. Intelligible models for healthcare: Pre- dicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, 1721- 1730. ACM.
- [12]12. Chen, M.; Zheng, A. X.; Lloyd, J.; Jordan, M. I.; and Brewer, E. 2004. Failure diagnosis using decision trees. In Au- tonomic Computing, 2004. Proceedings. International Con- ference on, 36-43. IEEE.
- [13]13. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei- Fei, L. 2009. Imagenet: A large-scale hierarchical im- age database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 248-255. IEEE.
- [14]14. Dietterich, T. G., and Horvitz, E. J. 2015. Rise of concerns about ai: reflections and directions. Communications of the ACM 58(10):38-40.
- [15]15. Fang, H.; Gupta, S.; Iandola, F. N.; Srivastava, R. K.; Deng, L.; Dollár, P.; Gao, J.; He, X.; Mitchell, M.; Platt, J. C.; Zitnick, C. L.; and Zweig, G. 2015. From captions to vi- sual concepts and back. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, 1473-1482.
- [17]17. Groce, A.; Kulesza, T.; Zhang, C.; Shamasunder, S.; Burnett, M.; Wong, W.-K.; Stumpf, S.; Das, S.; Shinsel, A.; Bice, F.; et al. 2014. You are the only possible oracle: Effec- tive test selection for end users of interactive machine learn- ing systems. IEEE Transactions on Software Engineering 40(3):307-323.
- [26]26. Lin, T.; Maire, M.; Belongie, S. J.; Hays, J.; Perona, P.; Ra- manan, D.; Dollár, P.; and Zitnick, C. L. 2014. Microsoft COCO: common objects in context. In Computer Vision - ECCV 2014 -13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V, 740-755.
- [28]28. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine transla- tion. In ACL, 311-318.
- [29]29. ACL.
- [30]30. Parikh, D., and Zitnick, C. L. 2011. Finding the weak- est link in person detectors. In Computer Vision and Pat- tern Recognition (CVPR), 2011 IEEE Conference on, 1425- 1432. IEEE.
- [39]39. Steinfeld, A.; Bennett, S. R.; Cunningham, K.; Lahut, M.; Quinones, P.-A.; Wexler, D.; Siewiorek, D.; Hayes, J.; Co- hen, P.; Fitzgerald, J.; et al. 2007. Evaluation of an inte- grated multi-task machine learning system with humans in the loop. In Proceedings of the 2007 Workshop on Perfor- mance Metrics for Intelligent Systems, 168-174. ACM.
- [44]44. Zhang, P.; Wang, J.; Farhadi, A.; Hebert, M.; and Parikh, D. 2014. Predicting failures of vision systems. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3566-3573.
Cited by
Sorted by citations
The role of artificial intelligence in achieving the Sustainable Development Goals
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., Felländer, A., Langhans, S. D., Tegmark, M., & Fuso Nerini, F. (2020) · Nature Communications · doi:10.1038/s41467-019-14108-y · Cited by 3,294 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800 · see citing works
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · see citing works
Self-citationNazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Singhal, A., Neveditsin, N., Tanveer, H., & Mago, V. (2024) · JMIR Medical Informatics · doi:10.2196/50048 · Cited by 172 · see citing works
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · see citing works
XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · see citing works
Improving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87 · see citing works
“More recently, crowd-based systems such as Pandora [44], Deblinder [9], and P-BTM [38] have shown how crowdsourcing can augment behavioral analysis tools by surfacing previously unknown model failures.”
BackgroundMethodDesigning Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72 · see citing works
Self-citationZeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581268 · Cited by 51 · see citing works
Color for Characters - Effects of Visual Explanations of AI on Trust and Observability
Schrills, T., & Franke, T. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-50334-5_8 · Cited by 17 · see citing works
Human-Centered Evaluation and Auditing of Language Models
Liu, Y. L., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Liao, Q. V., Xu, W., Novikova, J., & Xiao, Z. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706729 · Cited by 7 · see citing works
- Authors
- Nushi, B., Kamar, E., & Horvitz, E.
- Affiliations
- Microsoft (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
- Published
- 2018-06-15
- Volume
- 6
- Pages
- 126-135
- ISSN
- 2769-1330 (ISSN-L)
- 2769-1349
- OpenAlex
- W2810918175
- Cited by
- 126
- References
- 28
- Citations by year
- 2026: 10 · 2025: 13 · 2024: 14 · 2023: 19 · 2022: 21 · 2021: 25 · 2020: 11 · 2019: 12 · 2018: 1
- FWCI
- 4.98
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- National Conference on Artificial Intelligence
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- AI accountability, explainable AI, image captioning, human oversight, system reliability, Pandora
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
