The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Abstract
Machine learning is an important tool for decision making, but its ethical and responsible application requires rigorous vetting of its interpretability and utility: an understudied problem, particularly for natural language processing models. We propose an evaluation of interpretation on a real task with real human users, where the effectiveness of interpretation is measured by how much it improves human performance. We design a grounded, realistic human-computer cooperative setting using a question answering task, Quizbowl. We recruit both trivia experts and novices to play this game with computer as their teammate, who communicates its prediction via three different interpretations. We also provide design guidance for natural language processing human-in-the-loop settings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“researchers have discussed the importance of an AI explanation to communicate an AI output to a user [5, 20, 29] and realize human-AI collaboration [48] for a decision-making task [15, 42].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [20]
In the citing paper
“Such measures include, but are not limited to measuring end-users' decision time [19,62], how many times end-users requested AI recommendation [52], and how many times they agreed with AI recommendations [7,23,58].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [19]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294
Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01720.x · Cited by 1,180
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
International Journal of Human-Computer Studies
Sdol, E. (2009) · Cited by 1,095
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687
Ross, A., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11504 · Cited by 589
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Numeracy, Ratio Bias, and Denominator Neglect in Judgments of Risk and Probability
Reyna, V. F., & Brainerd, C. J. (2008) · Learning and Individual Differences · doi:10.1016/j.lindif.2007.03.011 · Cited by 481
Allen, J. E., Guinn, C. I., & Horvtz, E. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.796083 · Cited by 442
Semantically Equivalent Adversarial Rules for Debugging NLP models
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1079 · Cited by 432
Understanding Neural Networks through Representation Erasure
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 412
Towards better analysis of machine learning models: A visual analytics perspective
Liu, S., Wang, X., Liu, M., & Zhu, J. (2017) · Visual Informatics · doi:10.1016/j.visinf.2017.01.006 · Cited by 397
XPLAIN: a system for creating and explaining expert consulting programs
Swartout, W. R. (1983) · Artificial Intelligence · doi:10.1016/s0004-3702(83)80014-9 · Cited by 393
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
Krause, J., Perer, A., & Ng, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858529 · Cited by 390
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 335
A survey of software learnability: metrics, methodologies and guidelines
Grossman, T., Fitzmaurice, G., & Attar, R. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518803 · Cited by 322
A Neural Network for Factoid Question Answering over Paragraphs
Iyyer, M., Boyd-Graber, J., Claudino, L., Socher, R., & Daumé III, H. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1070 · Cited by 301
Self-citationRoss, A. S., & Doshi‐Velez, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09404 · Cited by 265
To Trust Or Not To Trust A Classifier
Jiang, H., Kim, B., Guan, M. Y., & Gupta, M. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11783 · Cited by 262
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257
Pathologies of Neural Models Make Interpretations Difficult
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., & Boyd-Graber, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1407 · Cited by 245
Self-citationBeyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMsPreprint
Murdoch, W. J., Liu, P. J., & Yu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05453 · Cited by 232
Explainable Artificial Intelligence (XAI)
Mohaghegh, S. D. (2024) · Artificial Intelligence for Science and Engineering Applications · doi:10.1201/9781003369356-8 · Cited by 215
Elasticsearch: The Definitive Guide
Gormley, C., & Tong, Z. (2015) · Cited by 189
New Potentials for Data‐Driven Intelligent Tutoring System Development and Optimization
Koedinger, K. R., Brunskill, E., Baker, R. S. J. D., McLaughlin, E. A., & Stamper, J. (2013) · AI Magazine · doi:10.1609/aimag.v34i3.2484 · Cited by 178 · cited 1×
“Applying this thinking to the evaluation, we measure interpretability by asking what ML can do for humans: a good interpretation is one that augments [30] human intelligence.”
MethodThe Design of Implicit Interactions: Making Interactive Systems Less Obnoxious
Ju, W., & Leifer, L. (2008) · Design Issues · doi:10.1162/desi.2008.24.3.72 · Cited by 177
Smarter than you think: how technology is changing our minds for the better
(2014) · Choice Reviews Online · doi:10.5860/choice.51-3207 · Cited by 157
Opponent Modeling in Deep Reinforcement LearningPreprint
He, H., Boyd-Graber, J. L., Kwok, K., & Daumé, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.05559 · Cited by 156
Self-citationNarayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153
Towards improving trust in context-aware systems by displaying system confidence
Antifakos, S., Kern, N., Schiele, B., & Schwaninger, A. (2005) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/1085777.1085780 · Cited by 124
The human touch: How non-expert users perceive, interpret, and fix topic models
Lee, T. Y., Smith, A., Seppi, K. D., Elmqvist, N., Boyd-Graber, J. L., & Findlater, L. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.03.007 · Cited by 102
Self-citationAI Challenges in Human-Robot Cognitive TeamingPreprint
Chakraborti, T., Subbarao Kambhampati, B., Scheutz, M. J., & Zhang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04775 · Cited by 100
Don’t Until the Final Verb Wait: Reinforcement Learning for Simultaneous Machine Translation
Grissom, A., He, H., Boyd-Graber, J. L., Morgan, J., & Daumé, H. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1140 · Cited by 99
Self-citationEvaluating Feature Importance EstimatesPreprint
Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2018) · arXiv (Cornell University) · Cited by 84
Intervention user interfaces: a new interaction paradigm for automated systems
Schmidt, A., & Herrmann, T. A. (2017) · interactions · doi:10.1145/3121357 · Cited by 81
How to Explain Individual Classification Decisions
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859912 · Cited by 54 · cited 1×
“y forms, and within each form we have multiple methods to generate the interpretation. For example, to highlight salient input features for image classification, we can use variants of input gradient [4, 60]. To optimize the generalizability of our results (despite being task-specific) and demonstrate the flexibility…”
MethodSupWSD: A Flexible Toolkit for Supervised Word Sense Disambiguation
Papandrea, S., Raganato, A., & Bovi, C. D. (2017) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d17-2018 · Cited by 47
Besting the Quiz Master: Crowdsourcing Incremental Classification Games
Boyd-Graber, J. L., Satinoff, B., He, H., & Daumé, H. (2012) · Empirical Methods in Natural Language Processing · Cited by 35
Self-citationVisualization of uncertainty in context aware mobile applications
Rukzio, E., Hamard, J., Noda, C., & Luca, A. D. (2006) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/1152215.1152267 · Cited by 33
Smith, A., Lee, T. Y., Poursabzi-Sangdeh, F., Boyd-Graber, J. L., Elmqvist, N., & Findlater, L. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00042 · Cited by 31
Self-citationImproving Human-Machine Cooperative Visual Search With Soft Highlighting
Kneusel, R. T., & Mozer, M. C. (2017) · ACM Transactions on Applied Perception · doi:10.1145/3129669 · Cited by 22
Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions
Wallace, E., & Boyd-Graber, J. (2018) · Proceedings of ACL 2018, Student Research Workshop · doi:10.18653/v1/p18-3018 · Cited by 14
Self-citationHuman-Computer Question Answering: The Case for Quizbowl
Boyd-Graber, J. L., Feng, S., & Rodríguez, P. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-94042-7_9 · Cited by 8
Self-citationVinson, D. W., Takayama, L. A., Forlizzi, J., Ju, W. G., Çakmak, M., & Kuzuoka, H. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3186326 · Cited by 6
Research works printed but not linked: 6
- Ian J. Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015 . Explaining and Harnessing Adversarial Examples . In Proceedings of the International Conference on Learning Representations. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In Proceedings of the International Conference on Learning Representations.
- Pieter-Jan Kindermans , Sara Hooker , Julius Adebayo , Maximilian Alber , Kristof T. Schütt , Sven Dähne , Dumitru Erhan , and Been Kim . 2017. The (Un)reliability of saliency methods. arXiv preprint arXiv: 1711.00867 ( 2017 ). Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2017. The (Un)reliability of saliency methods. arXiv preprint arXiv: 1711.00867 (2017). · arXiv
- Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. arXiv preprint arXiv: 1606.03490 ( 2016 ). Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. arXiv preprint arXiv: 1606.03490 (2016). · arXiv
- Tim Miller . 2017. Explanation in artificial intelligence: insights from the social sciences. arXiv preprint arXiv:1706.07269 ( 2017 ). Tim Miller. 2017. Explanation in artificial intelligence: insights from the social sciences. arXiv preprint arXiv:1706.07269 (2017). · arXiv
- Cynthia Rudin . 2018. Please Stop Explaining Black Box Models for High Stakes Decisions. arXiv preprint arXiv:1811.10154 ( 2018 ). Cynthia Rudin. 2018. Please Stop Explaining Black Box Models for High Stakes Decisions. arXiv preprint arXiv:1811.10154 (2018). · arXiv
- USACM. 2017. Statement on algorithmic transparency and accountability . Public Policy Council ( 2017 ). USACM. 2017. Statement on algorithmic transparency and accountability. Public Policy Council (2017).
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“[17], experts and novices played Quiz Bowl with recommendation from Elastic Search system.”
BackgroundMethodInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466
“However, while explanations help with understanding, it is not yet clear under which conditions they improve collaboration and human productivity (Doshi-Velez and Kim 2017; Poursabzi-Sangdeh et al. 2018; Feng and Boyd-Graber 2019).”
BackgroundExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Metadata
- Authors
- Feng, S., & Boyd-Graber, J.
- Affiliations
- University of Maryland, College Park (US)
- Venue
- Proceedings of the 24th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2019-03-17
- Pages
- 229-239
- OpenAlex
- W2896487960
- Cited by
- 92
- References
- 68
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 14 · 2023: 24 · 2022: 9 · 2021: 15 · 2020: 11 · 2019: 5
- FWCI
- 3.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- National Science Foundation
- Raytheon Company
- Raytheon Technologies
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- interpretability, explainable AI, human-computer collaboration, question answering, human-in-the-loop, natural language processing models, ethical AI, human performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.