Crowdsourcing and Evaluating Concept-driven Explanations of Machine Learning Models
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
An important challenge in building explainable artificially intelligent (AI) systems is designing interpretable explanations. AI models often use low-level data features which may be hard for humans to interpret. Recent research suggests that situating machine decisions in abstract, human understandable concepts can help. However, it is challenging to determine the right level of conceptual mapping. In this research, we explore granularity (of data features) and context (of data instances) as dimensions underpinning conceptual mappings. Based on these measures, we explore strategies for designing explanations in classification models. We introduce an end-to-end concept elicitation pipeline that supports gathering high-level concepts for a given data set. Through crowd-sourced experiments, we examine how providing conceptual information shapes the effectiveness of explanations, finding that a balance between coarse and fine-grained explanations help users better estimate model predictions. We organize our findings into systematic themes that can inform design considerations for future systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“First, AI thought elicitation demands a good balance between human information needs and interpretability [1, 96].”
From Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2025) · cites this paper as [96]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 2×
“As model explanations reach broader audiences (both because ML is increasingly being integrated into everyday life [20] and there is increasing societal focus on model accountability [24]), researchers have explored new, more accessible techniques for explaining complex models.”
BackgroundDescribing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · cited 1×
“For instance, in [4], densely annotated datasets like Pascal-Part[12], Pascal-Context[44] and Describable Textures[14] were used to assemble a dataset with pixel-level labels of each image describing colors, materials, textures, parts, objects and scenes.”
MethodNetwork Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · cited 1×
“This has motivated recent research in explainable AI[27] (xAI), with one main thread focusing on visualizing model properties or data features that are crucial in completing the task.”
BackgroundJiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729
What do we need to build explainable AI systems for the medical domain?
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017) · arXiv · doi:10.48550/arxiv.1712.09923 · Cited by 636 · cited 1×
“In order to work efficiently with their human counterparts in critical decision-making activities such as healthcare [29], finance [58], and security [40] while remaining transparent and accountable, Machine Leaning (ML) models should not only be accurate but also must explain their decisions effectively.”
BackgroundDetect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.254 · Cited by 605
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225 · cited 1×
“Some metrics include users’ trust and confidence [28], their satisfaction [6], and verbalizations of their mental representations of the model [36].”
BackgroundEvaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163
Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153
Group emotion recognition with individual facial emotion CNNs and global image based CNNs
Tan, L., Zhang, K., Wang, K., Zeng, X., Peng, X., & Qiao, Y. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143008 · Cited by 73 · cited 1×
“Though image classification of animals and objects, as examined in this paper, may seem trivial, the methods can be transferred to practical applications like agricultural auditing [56], and facial recognition [64].”
Background
Research works printed but not linked: 58
- 10.1109/TPAMI.2012.120 · DOI · Paper page
- 10.1145/3097983.3098070 · DOI · Paper page
- Ashwin Bhandare, Maithili Bhide, Pranav Gokhale, and Rohan Chandavarkar. 2016. Applications of convolutional neural networks. International Journal of Computer Science and Information Technologies, Vol. 7, 5 (2016), 2206--2215.
- Mustafa Bilgic and Raymond J. Mooney. 2005. Explaining Recommendations: Satisfaction vs. Promotion. In Proceedings of Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research at the 2005 International Conference on Intelligent User Interfaces, Vol. 5. IUI, San Diego, CA, 153. http://www.cs.utexas.edu/users/ai-lab?bilgic:iui-bp05
- 10.1007/978-981-10-0557-2_87 · DOI · Paper page
- 10.5555/1866696.1866697 · doi:10.5555/1866696.1866697
- 10.1017/S0140525X10000919 · DOI · Paper page
- 10.1145/2818048.2820023 · DOI · Paper page
- 10.1145/3025453.3026044 · DOI · Paper page
- 10.1145/2470654.2466265 · DOI · Paper page
- 10.18653/v1 · doi:10.18653/v1
- 10.1109/CVPR.2013.81 · DOI · Paper page
- 10.1145/1015330.1015408 · DOI · Paper page
- Anthony Elliott. 2019. The culture of AI: Everyday life and the digital revolution .Routledge, Australia.
- 10.1145/2702123.2702556 · DOI · Paper page
- 10.5555/1866696.1866709 · doi:10.5555/1866696.1866709
- Nils Gehlenborg and Bang Wong. 2012. Points of view: heat maps. · DOI
- 10.1109/CVPR.2014.81 · DOI · Paper page
- 10.5120/11638-7118 · DOI · Paper page
- 10.1145/2856767.2856811 · DOI · Paper page
- 10.1145/1807342.1807376 · DOI · Paper page
- Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications.
- Minsuk Kahng, Nikhil Thorat, Duen Horng Polo Chau, Fernanda B Viégas, and Martin Wattenberg. 2018. Gan lab: Understanding complex deep generative models using interactive visual experimentation. IEEE transactions on visualization and computer graphics, Vol. 25, 1 (2018), 1--11.
- Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, Stockholmsmässan, Stockholm Sweden, 2668--2677. http://proceedings.mlr.press/v80/kim18d.html
- 10.1145/1357054.1357127 · DOI · Paper page
- 10.1109/WACV.2007.18 · DOI
- N. Luhmann H. Davis J. Raffan K. Rooney M. King and C. Morgner. 1979. Trust and Power. Wiley USA.
- Scott M. Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 4768--4777.
- 10.1109/IECON.2018.8591457 · DOI · Paper page
- 10.2307/258792 · DOI · Paper page
- Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2020. A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems. arxiv: cs.HC/1811.11839
- Gregory Murphy. 2004. The big book of concepts. MIT press, USA.
- 10.18653/v1 · doi:10.18653/v1
- David Oleson, Alexander Sorokin, Greg Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. In Proceedings of the 11th AAAI Conference on Human Computation (AAAIWS'11-11). AAAI Press, San Francisco, USA, 43--48.
- Forough Poursabzi-Sangdeh, Dan Goldstein, Jake Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018a. Manipulating and Measuring Model Interpretability. https://www.microsoft.com/en-us/research/publication/manipulating-and-measuring-model-interpretability/
- Forough Poursabzi-Sangdeh, Dan Goldstein, Jake Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018b. Manipulating and Measuring Model Interpretability. https://www.microsoft.com/en-us/research/publication/manipulating-and-measuring-model-interpretability/
- Daryl Pregibon et almbox. 1981. Logistic regression diagnostics. The Annals of Statistics, Vol. 9, 4 (1981), 705--724.
- J. Ross Quinlan. 1986. Induction of decision trees. Machine learning, Vol. 1, 1 (1986), 81--106.
- 10.1145/2939672.2939778 · DOI · Paper page
- Eleanor H Rosch. 1973. Natural categories. Cognitive psychology, Vol. 4, 3 (1973), 328--350.
- Denise M Rousseau, Sim B Sitkin, Ronald S Burt, and Colin Camerer. 1998. Not so different after all: A cross-discipline view of trust. Academy of management review, Vol. 23, 3 (1998), 393--404.
- 10.1145/2047196.2047199 · DOI · Paper page
- 10.1016/j.eswa.2019.113100 · DOI · Paper page
- Philipp Schmidt and Felix Biessmann. 2019. Quantifying Interpretability and Trust in Machine Learning Systems. arxiv: cs.LG/1901.08558
- 10.1109/ICCV.2017.74 · DOI · Paper page
- 10.1109/ICCV.2019.00268 · DOI · Paper page
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arxiv: cs.CV/1409.1556
- 10.1145/2968219.2968326 · DOI
- 10.1109/CVPR.2015.7298935 · DOI · Paper page
- 10.1108/00330330610681295 · DOI · Paper page
- Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems, Vol. 2, 1--3 (1987), 37--52.
- 10.3115/981732.981751 · DOI · Paper page
- Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and Understanding Convolutional Networks. In Computer Vision -- ECCV 2014,, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 818--833.
- 10.1631/FITEE.1700808 · DOI · Paper page
- Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. 2018a. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, Vol. 41, 9 (2018), 2131--2145.
- 10.1109/CVPR.2016.319 · DOI · Paper page
- 10.1007/978-3-030-01237-3_8 · DOI · Paper page
- Radim ?eh??ek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. pam??ový nosi?. In Proceedings of LREC 2010 workshop New Challenges for NLP Frameworks (Valletta, Malta). University of Malta, Valletta, Malta, 46--50. http://www.fi.muni.cz/usr/sojka/presentations/lrec2010-poster-rehurek-sojka.pdf
Cited by
Silva, A., Schrum, M., Hedlund-Botti, E., Gopalan, N., & Gombolay, M. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2101698 · Cited by 104
“Often, this work involves gathering a large corpus of annotated explanations or images, which can then be leveraged to learn a generative language model that produces natural-language sequences to explain given model input samples and output predictions (Mishra & Rzeszotarski, 2021).”
BackgroundInvestigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
“A related approach is to support the interpretability of model mechanisms [37, 43, 53], increasing the predictability [18, 21] of AI behavior.”
BackgroundMa, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713423 · Cited by 48
“First, AI thought elicitation demands a good balance between human information needs and interpretability [1, 101].”
Anthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713329 · Cited by 24
“Researchers have begun to empirically test particular XAI techniques to improve human-AI performance, such as behavior descriptions [22], cognitive forcing functions [20], and balancing different levels (e.g., fine-grained, coarse-grained) of conceptual information [118].”
Anthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2024) · arXiv · doi:10.48550/arxiv.2407.08867 · Cited by 24
Metadata
- Authors
- Mishra, S., & Rzeszotarski, J. M.
- Affiliations
- Cornell University, Ithaca, NY, USA
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-26
- ISSN
- 2573-0142
- Cited by
- 21
- References
- 13
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.