Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Suresh, Harini; Gomez, Steven R.; Nam, Kevin K.; Satyanarayan, Arvind

doi:10.1145/3411764.3445088

Cited by 119Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISoftware Testing and Debugging Techniques

Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Open access

Suresh, Harini, Gomez, Steven R., Nam, Kevin K., Satyanarayan, Arvind·2021·Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

DOI 10.1145/3411764.3445088

Open this paper in CoChat — free account Open PDF

Abstract

To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders’ knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1145/3411764.3445088

Cites

82 of 86 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Recent HCI work has begun to pursue integrating metacognitive support into GenAI systems, albeit without explicitly grounding it in metacognition.
“Recent HCI work has begun to pursue this direction [173,204], albeit without explicitly grounding it in metacognition; we suggest that a metacognitive lens offers new research and design directions for augmenting GenAI system usability.”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [173]
Section 3.2 describes helping users understand a model’s limitations and decide how to incorporate its output into further actions, which is related to the claimed direction. But the paper addresses interpretable machine learning, not integrating metacognitive support into GenAI systems; it does not substantiate that more specific attribution. (LLM, conf=0.76)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1× · see citing works
“In this case, instead of starting with a tabula rasa, perhaps an interpretability interface would be initialized with summaries of the model’s strengths and weaknesses (akin to a model card [84]).”
Method
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works
Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743 · see citing works
Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 1,372 · cited 6× · see citing works
““anticipate ethics-related failures before launch" [97], bias or mistake detection [89, 101, 103, 125], understand skewness and biases in input data [35], find unknown vulnerabilities and flaws [5, 38] (8/58) T3: Understand the extent of the information the model is using data entanglement [97], be…”
BackgroundInfluential
Why and Where: A Characterization of Data Provenance
Buneman, P., Khanna, S., & Wang-Chiew, T. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44503-x_20 · Cited by 1,334 · see citing works
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268 · see citing works
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061 · see citing works
Introduction to Action Research
Greenwood, D. J., & Levin, M. (2007) · Sage eBooks · doi:10.4135/9781412984614 · Cited by 1,052 · see citing works
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · see citing works
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · see citing works
Machine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912 · see citing works
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912 · see citing works
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · see citing works
Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Feature visualization
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880 · see citing works
Interpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833 · see citing works
A Multi-Level Typology of Abstract Visualization Tasks
Brehmer, M., & Munzner, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.124 · Cited by 745 · see citing works
Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · see citing works
Community-based participatory research and integrated knowledge translation: advancing the co-creation of knowledge
Jull, J., Giles, A., & Graham, I. D. (2017) · Implementation Science · doi:10.1186/s13012-017-0696-3 · Cited by 651 · see citing works
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633 · see citing works
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 3× · see citing works
“users [68, 136], “actors bring their own points of view and own priorities" [125], “people employ certain biases and social expectations" [83], “anticipating the situated, user-encountered capability of AI is difficult" [127], familiarity with privacy and personal data issues [44], individual fatigue and workflow issu…”
BackgroundInfluential
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615 · see citing works
"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · see citing works
The Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591 · see citing works
Self-citation
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · see citing works
<i>Why and why not</i> explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · see citing works
Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Gao, L., Zhu, X., Song, J., Zhao, Z., & Shen, H. T. (2019) · doi:10.24963/ijcai.2019 · Cited by 538 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 6× · see citing works
“The next-most frequent style of evaluation are user studies conducted with proxy stakeholders (e.g., sourced from Amazon Mechanical Turk or a similar online platform) and/or proxy tasks (e.g., guessing a model’s outcome) [26, 29, 37, 71, 79, 95, 130].”
BackgroundInfluential
Untitled work
(2014) · Digital Commons at University of Maryland Carey Law (University of Maryland Francis King Carey School of Law) · Cited by 524 · see citing works
Data colonialism through accumulation by dispossession: New metaphors for daily data
Thatcher, J., O’Sullivan, D., & Mahmoudi, D. (2016) · Environment and Planning D: Society and Space · doi:10.1177/0263775816633195 · Cited by 512 · see citing works
The What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., WATTENBERG, M. P., Viégas, F. B., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 512 · see citing works
Actionable Recourse in Linear Classification
Ustun, B., Spangher, A., & Liu, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287566 · Cited by 501 · see citing works
Accountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457 · cited 3× · see citing works
“The ability to detect discrimination or other unwanted logic codifed in a model is considered a crucial tool for being able to contest an automated decision (O7) [39].”
BackgroundMethod
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · see citing works
A systematic review and taxonomy of explanations in decision support and recommender systems
Nunes, I., & Jannach, D. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9195-0 · Cited by 416 · see citing works
Designing interaction, not interfaces
Beaudouin-Lafon, M. (2004) · Working Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/989863.989865 · Cited by 410 · see citing works
Critical Race Theory for HCI
Ogbonnaya-Ogburu, I. F., Smith, A. D. R., To, A., & Toyama, K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376392 · Cited by 404 · see citing works
On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 7× · see citing works
“…[80], Tomsett et al. [113]), evaluations of specific interpretability techniques (e.g., Balog and Radlinski [10], Cai et al. [26], Cheng et al. [29]) and user/case studies that provide insights into pertinent human factors in interpretability (e.g., Hong et al. [58], Liao et al. [76],…”
BackgroundInfluential
Explaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286 · see citing works
Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning
Byrne, R. M. J. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/876 · Cited by 285 · see citing works
The work that visualisation conventions do
Kennedy, H., Hill, R. L., Aiello, G., & Allen, W. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1153126 · Cited by 278 · see citing works
On the Nature of Teaching and Teacher Education
Labaree, D. F. (2000) · Journal of Teacher Education · doi:10.1177/0022487100051003011 · Cited by 275 · see citing works
Strangers at the Gate: Gaining Access, Building Rapport, and Co-Constructing Community-Based Research
Le Dantec, C. A., & Fox, S. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675147 · Cited by 255 · see citing works
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Ras, G., van Gerven, M., & Haselager, P. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_2 · Cited by 255 · see citing works
Assessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246 · see citing works
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241 · see citing works
DeepRED – Rule Extraction from Deep Neural Networks
Zilke, J. R., Mencía, E. L., & Janssen, F. W. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46307-0_29 · Cited by 236 · see citing works
Gamut
Hohman, F., Head, A., Caruana, R., DeLine, R. A., & Drucker, S. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225 · see citing works
The effects of example-based explanations in a machine learning interface
Cai, C. J., Jongejan, J., & Holbrook, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302289 · Cited by 220 · see citing works
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219 · see citing works
Toward establishing trust in adaptive agents
Glass, A., McGuinness, D. L., & Wolverton, M. J. (2008) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1378773.1378804 · Cited by 217 · see citing works
Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188 · see citing works
Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · see citing works
Präferenz für Intuition und Deliberation (PID)
Betsch, C. (2004) · Zeitschrift für Differentielle und Diagnostische Psychologie · doi:10.1024/0170-1789.25.4.179 · Cited by 164 · see citing works
Data is Personal
Peck, E. M., Ayuso, S. E., & El-Etr, O. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300474 · Cited by 156 · see citing works
Designing and implementing transparency for real time inspection of autonomous robots
Theodorou, A. A., Wortham, R. H., & Bryson, J. J. (2017) · Connection Science · doi:10.1080/09540091.2017.1310182 · Cited by 154 · see citing works
Explainability scenarios: towards scenario-based XAI design
Wolf, C. T. (2019) · International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302317 · Cited by 153 · see citing works
Ethical Dimensions of Visualization Research
Correll, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300418 · Cited by 150 · see citing works
Toolkit to support intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2010) · Proceedings of the 12th ACM international conference on Ubiquitous computing · doi:10.1145/1864349.1864353 · Cited by 143 · see citing works
How it works
Tullio, J., Dey, A. K., Chalecki, J., & Fogarty, J. A. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240630 · Cited by 137 · see citing works
Studying up: reorienting the study of algorithmic fairness around issues of power
Barabas, C., Doyle, C., Rubinovitz, J. B., & Dinakar, K. (2020) · FAT* · doi:10.1145/3351095.3372859 · Cited by 135 · see citing works
Frames and Slants in Titles of Visualizations on Controversial Topics
Kong, H., Liu, Z., & Karahalios, K. G. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174012 · Cited by 123 · see citing works
Learning Through Practice
Billett, S. R. (2010) · Springer eBooks · doi:10.1007/978-90-481-3939-2 · Cited by 110 · see citing works
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96 · see citing works
Trust and Recall of Information across Varying Degrees of Title-Visualization Misalignment
Kong, H., Liu, Z., & Karahalios, K. G. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300576 · Cited by 96 · see citing works
Measuring Recommendation Explanation Quality
Balog, K., & Radlinski, F. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401032 · Cited by 91 · see citing works
The Dawn of a Critical Transparency Right for the Profiling Era
Hildebrandt, M. (2012) · IOS Press eBooks · doi:10.3233/978-1-61499-057-4-41 · Cited by 86 · see citing works
What Are People Doing About XAI User Experience? A Survey on AI Explainability Research and Practice
Ferreira, J. J., & Monteiro, M. D. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49760-6_4 · Cited by 83 · see citing works
Reconsidering the dimensions of expertise: from linear stages towards dual processing
Kinchin, I. M., & Cabot, B. (2010) · London Review of Education · doi:10.1080/14748460.2010.487334 · Cited by 77 · see citing works
What Do People Really Want When They Say They Want "Explainable AI?" We Asked 60 Stakeholders.
Brennen, A. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3383047 · Cited by 76 · see citing works
Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making
Suresh, H., Lao, N., & Liccardi, I. (2020) · 12th ACM Conference on Web Science · doi:10.1145/3394231.3397922 · Cited by 49 · see citing works
Self-citation
An integrated model of professional expertise and its implications for higher education
Yielder, J. (2004) · International Journal of Lifelong Education · doi:10.1080/0260137032000172060 · Cited by 45 · see citing works
A user-based taxonomy for deep learning visualization
Yu, R., & Shi, L. (2018) · Visual Informatics · doi:10.1016/j.visinf.2018.09.001 · Cited by 34 · see citing works
Broadening Intellectual Diversity in Visualization Research Papers
Lee, B., Isaacs, K. E., Szafir, D. A., Marai, G. E., Turkay, Ç., Tory, M. K., Carpendale, S., & Endert, A. (2019) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2019.2914844 · Cited by 33 · see citing works
An Exploratory Study on Visual Exploration of Model Simulations by Multiple Types of Experts
Boukhelifa, N., Bezerianos, A., Trelea, I.-C., Perrot, N., & Lutton, É. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300874 · Cited by 25 · see citing works
Exploring Expertise
(1998) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-13693-3 · Cited by 18 · see citing works
Tell Me What You Know
Monteiro-Krebs, L., Alvarado, O., Dewitte, P., Ausloos, J., Geerts, D., Naudts, L., & Verbert, K. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312808 · Cited by 16 · see citing works
Explainable Clustering Using Hyper-Rectangles for Building Energy Simulation Data
Bhatia, A., Garg, V., Haves, P., & Pudi, V. (2019) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/238/1/012068 · Cited by 11 · see citing works

Flagged first, then by citations

Research works printed but not linked: 50

2020. ML Interpretability for Scientifc Discovery (MLI4SD) Workshop. https: //sites.google.com/view/mli4sd-icml2020/home. Accessed: 2020-09-16.
Agnar Aamodt and Enric Plaza. 1994. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI communications 7, 1 (1994), 39-59.
Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hofman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varsh- ney, Dennis Wei, and Yunfeng Zhang. 2019. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. arXiv:1909.03012 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1909.03012 arXiv: 1909.03012. · arXiv
Robert K Atkinson, Sharon J Derry, Alexander Renkl, and Donald Wortham. 2000. Learning from examples: Instructional principles from the worked examples research. Review of educational research 70, 2 (2000), 181-214.
Krisztian Balog and Filip Radlinski. 2020. Measuring Recommendation Explana- tion Quality:The Conficting Goals of Explanations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '20). ACM, New York, NY, USA, Virtual Event, 10.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S. Weld. 2020. Does the Whole Exceed its Parts? The Efect of AI Explanations on Complementary Team Per- formance. arXiv:2006.14779 [cs] (June 2020). http://arxiv.org/abs/2006.14779 arXiv: 2006.14779. · arXiv
Hubert L Dreyfus and Stuart E Dreyfus. 1986. The power of human intuition //doi.org/10.1145/3351095.3375624 and expertise in the era of the computer. Mind over machine. Nueva York: The · DOI · Paper page
Nadia Boukhelifa, Anastasia Bezerianos, Ioan Cristian Trelea, Nathalie Méjean the digital humanities 150 (2012), 85-95.
Juliana J. Ferreira and Mateus S. Monteiro. 2020. What Are People Doing About visualization tasks. IEEE transactions on visualization and computer graphics 19, XAI User Experience? A Survey on AI Explainability Research and Practice. 12 (2013), 2376-2385.
James Fleck. 1998. Expertise: knowledge, power and tradeability. In Exploring //doi.org/10.1145/3334480.3383047 expertise. Springer, 143-171. · DOI
Ruth C. Fong and Andrea Vedaldi. 2017. Interpretable Explanations of Black A characterization of data provenance. In International conference on database Boxes by Meaningful Perturbation. In Proceedings of the IEEE International theory. Springer, 316-330. Conference on Computer Vision (ICCV).
Donna Haraway. 1988. Situated knowledges: The science question in feminism International Conference on Intelligent User Interfaces. 258-262. and the privilege of partial perspective. Feminist studies 14, 3 (1988), 575-599.
Johanna Hartelius. 2011. Rhetorics of Expertise. Social Epistemology 25, 3 Human-Computer Interaction 3, CSCW (Nov. 2019), 1-24. https://doi.org/10. (July 2011), 211-215. https://doi.org/10.1080/02691728.2011.578301 Publisher: 1145/3359206 Routledge _eprint: https://doi.org/10.1080/02691728.2011.578301. · DOI
Jonathan K Su. 2019. This looks like that: deep learning for interpretable image What Do You Need to Know?. In Augmented Cognition. Theoretical and Techno- recognition. In Advances in neural information processing systems. 8930-8941.
logical Approaches, Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.). Springer
Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M. Dark Patterns of Explainability, Transparency, and User Control for Intelligent Drucker. 2019. Gamut: A Design Probe to Understand How Data Scientists Systems. Los Angeles (2019), 6. Understand Machine Learning Models. In Proceedings of the 2019 CHI Conference
Danielle Keats Citron and Frank Pasquale. 2014. The scored society: Due process on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19).
Amanda Cox. 2011. Shaping Data for News. https://vimeo.com/29391942 IEEE Transactions on Visualization and Computer Graphics 25, 8 (Aug. 2019),
A critical review of stage models. Review of educational research 76, 3 (2006),
Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. 2020. Human Factors in 383-412. Model Interpretability: Industry Practices, Challenges, and Needs. Proceedings
Amit Dhurandhar, Vijay Iyengar, Ronny Luss, and Karthikeyan Shanmugam. efects in narrative visualization. IEEE transactions on visualization and computer 2017. A formal framework to characterize interpretability of procedures. (2017). graphics 17, 12 (2011), 2231-2240. arXiv:1707.03886. · arXiv
Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of and Jennifer Wortman Vaughan. 2020. Interpreting Interpretability: Under- Interpretable Machine Learning. arXiv:1702.08608 [cs, stat] (March 2017). standing Data Scientists' Use of Interpretability Tools for Machine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. · arXiv
Sina Mohseni, Jeremy E. Block, and Eric D. Ragan. 2020. A Human- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Grounded Evaluation Benchmark for Local Explanations of Machine Learn- 1-13. ing. arXiv:1801.05075 [cs] (June 2020). http://arxiv.org/abs/1801.05075 arXiv: · arXiv
Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2020. A Multidisciplinary you know: GDPR implications on designing transparency and accountability for Survey and Framework for Design and Evaluation of Explainable AI Systems. news recommender systems. In Extended Abstracts of the 2019 CHI Conference arXiv:1811.11839 [cs] (April 2020). http://arxiv.org/abs/1811.11839 arXiv: on Human Factors in Computing Systems. 1-6. 1811.11839. · arXiv
nale Doshi-Velez. 2018. How do humans understand explanations from machine International Conference on Machine Learning Workshop. learning systems? an evaluation of the human-interpretability of explanation.
Ingrid Nunes and Dietmar Jannach. 2017. A systematic review and taxonomy
Isaac Lage, Emily Chen, Jefrey He, Menaka Narayanan, Been Kim, Samuel J. of explanations in decision support and recommender systems. User Modeling Gershman, and Finale Doshi-Velez. 2019. Human Evaluation of Models Built and User-Adapted Interaction 27, 3-5 (2017), 393-444.
for Interpretability. Proceedings of the AAAI Conference on Human Computation
Ihudiya Finda Ogbonnaya-Ogburu, Angela D.R. Smith, Alexandra To, and Ken- and Crowdsourcing 7, 1 (Oct. 2019), 59-67. https://www.aaai.org/ojs/index.php/ taro Toyama. 2020. Critical Race Theory for HCI. In Proceedings of the 2020 CHI HCOMP/article/view/5280 Number: 1. Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA,
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, 10.1145/3287560.3287590 · DOI · Paper page
Himabindu Lakkaraju and Osbert Bastani. 2020. "How do I fool you?": Manip- pretability. Distill 3, 3 (2018), e10.
ulating User Trust via Misleading Black Box Explanations. In Proceedings of
Christopher A. Le Dantec and Sarah Fox. 2015. Strangers at the Gate: Gaining and perceptions of data visualization in rural pennsylvania. In Proceedings of Access, Building Rapport, and Co-Constructing Community-Based Research. In the 2019 CHI Conference on Human Factors in Computing Systems. 1-12. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jen- & Social Computing -CSCW '15. ACM Press, Vancouver, BC, Canada, 1348-1358.
Bongshin Lee, Kate Isaacs, Danielle Albers Szafr, G Elisabeta Marai, Cagatay 1802.07810 arXiv: 1802.07810. · arXiv
Alexander Renkl. 2014. Toward an instructionally oriented theory of example- Ubiquitous Computing (Copenhagen, Denmark) (UbiComp '10). Association for based learning. Cognitive science 38, 1 (2014), 1-37.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Model-agnostic 1864349.1864353 interpretability of machine learning. (2016). arXiv:1606.05386. · DOI · Paper page
Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A Explanations Improve the Intelligibility of Context-Aware Intelligent Systems. proposal of user-centered explainable AI.. In IUI Workshops. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems [102]
Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM
Keim. 2019. Towards a Rigorous Evaluation of XAI Methods on Time Se- conference on human factors in computing systems. 1-15. ries. arXiv:1909.07082 [cs] (Sept. 2019). http://arxiv.org/abs/1909.07082 arXiv: · arXiv
Ryan Weber. 2012. Review of The Rhetoric of Expertise. Rhetoric and Public 1909.07082. Afairs 15, 1 (2012), 193-196. https://www.jstor.org/stable/41955617 Publisher:
Johannes Schneider and Joshua Handali. 2019. PERSONALIZED EXPLANATION Michigan State University Press.
FOR MACHINE LEARNING: A CONCEPTUALIZATION. In Proceedings of the [121]
Adrian Weller. 2019. Transparency: Motivations and Challenges. 27th European Conference on Information Systems (ECIS). Stockholm & Uppsala, arXiv:1708.01870 [cs] (Aug. 2019). http://arxiv.org/abs/1708.01870 arXiv: Sweden. https://aisel.aisnet.org/ecis2019_rp/171 1708.01870. · arXiv
Jacob O Wobbrock and Julie A Kientz. 2016. Research contributions in human- Name: IEEE Transactions on Visualization and Computer Graphics. computer interaction. interactions 23, 3 (2016), 38-44.
Christine T. Wolf. 2019. Explainability scenarios: towards scenario-based XAI communication 52, 2 (2005), 163-174. design. In Proceedings of the 24th International Conference on Intelligent User
Harini Suresh, Natalie Lao, and Ilaria Liccardi. 2020. Misplaced Trust: Measuring Explainable Intelligent Systems for Medical Diagnosis. arXiv:1902.06019 [cs.HC] the Interference of Machine Learning in Human Decision-Making. In WebSci · arXiv
Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re- '20: 12th ACM Conference on Web Science, Southampton, UK, July 6-10, 2020, examining Whether, Why, and How Human-AI Interaction Is Uniquely Difcult
Sana Tonekaboni, Shalmali Joshi, Melissa D. McCradden, and Anna Goldenberg. //doi.org/10.1145/3290605.3300509 2019. What Clinicians Want: Contextualizing Explainable Machine Learning [131 · DOI · Paper page
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. 2018. Interpretable Con- Linear Classifcation. In Proceedings of the Conference on Fairness, Accountability, volutional Neural Networks. In Proceedings of the IEEE Conference on Computer and Transparency (Atlanta, GA, USA) (FAT* '19). Association for Computing Vision and Pattern Recognition (CVPR).

Sorted by citations

Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300 · see citing works
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 280 · see citing works
“Planning is a task-oriented metacognitive strategy entailing both the definition of clear goals (i.e., self-awareness) and devising a comprehensive approach for achieving them by breaking them down into smaller, manageable steps (i.e., task decomposition) [21].”
Background
Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI
Pushkarna, M., Zaldivar, A., & Kjartansson, O. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533231 · Cited by 231 · see citing works
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228 · see citing works
Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 196 · see citing works
From explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180 · see citing works
The coming of age of interpretable and explainable machine learning models
Lisboa, P. J. G., Saralajew, S., Vellido, A., Fernández-Domenech, R., & Villmann, T. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.02.040 · Cited by 155 · see citing works
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 145 · see citing works
The role of explainable AI in the context of the AI Act
Panigutti, C., Hamon, R., Hupont, I., Llorca, D. F., Yela, D. F., Junklewitz, H., Scalzo, S., Mazzini, G., Sánchez, I., Garrido, J. S., et al. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594069 · Cited by 171 · see citing works
Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143 · see citing works
Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care
Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141 · see citing works
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Liao, Q. V., Subramonyam, H., Wang, J., & Vaughan, J. W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 97 · see citing works
Disentangling Fairness Perceptions in Algorithmic Decision-Making: the Effects of Explanations, Human Oversight, and Contestability
Yurrita, M., Draws, T., Balayn, A., Murray-Rust, D., Tintarev, N., & Bozzon, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581161 · Cited by 73 · see citing works
“Such strategies should be adapted to decision subjects’ insight needs [88] and designed in a way that they would understand [11, 52].”
Background
Fairness via Explanation Quality
Dai, J. C., Upadhyay, S., Aïvodji, U., Bach, S. H., & Lakkaraju, H. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534159 · Cited by 62 · see citing works
Exploring the effects of human-centered AI explanations on trust and reliance
Scharowski, N., Perrig, S. A. C., Svab, M., Opwis, K., & Brühlmann, F. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1151150 · Cited by 57 · see citing works
Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work
Zheng, C., Wang, D., Wang, A. Y., & Ma, X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517615 · Cited by 54 · see citing works
Selective Explanations: Leveraging Human Input to Align Explainable AI
Lai, V. W.-M., Zhang, Y., Chen, C., Liao, Q. V., & Tan, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610206 · Cited by 64 · see citing works
Explainable reinforcement learning (XRL): a systematic literature review and taxonomy
Bekkemoen, Y. (2023) · Machine Learning · doi:10.1007/s10994-023-06479-7 · Cited by 58 · see citing works
Mitigating knowledge imbalance in AI-advised decision-making through collaborative user involvement
Gómez, C., Unberath, M., & Huang, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102977 · Cited by 55 · see citing works
Machine Learning Security in Industry: A Quantitative Survey
Grosse, K., Bieringer, L., Besold, T. R., Biggio, B., & Krombholz, K. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3251842 · Cited by 50 · see citing works
What Are the Users’ Needs? Design of a User-Centered Explainable Artificial Intelligence Diagnostic System
He, X., Hong, Y., Xi, Z., & Zhang, Y. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2095093 · Cited by 48 · see citing works
How mature is requirements engineering for AI-based systems? A systematic mapping study on practices, challenges, and future research directions
Habiba, U.-E., Haug, M., Bogner, J., & Wagner, S. (2024) · Requirements Engineering · doi:10.1007/s00766-024-00432-3 · Cited by 51 · see citing works
Towards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533118 · Cited by 42 · see citing works
Seamful XAI: Operationalizing Seamful Design in Explainable AI
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637396 · Cited by 61 · see citing works
Slide4N: Creating Presentation Slides from Computational Notebooks with Human-AI Collaboration
Wang, F., Liu, X., Liu, O., Neshati, A., Ma, T., Zhu, M., & Zhao, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580753 · Cited by 38 · see citing works
Human‐centered explainable artificial intelligence: An Annual Review of Information Science and Technology (ARIST) paper
Ridley, M. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24889 · Cited by 51 · see citing works
Knowledge Graphs in Practice: Characterizing their Users, Challenges, and Visualization Opportunities
Li, H. X., Appleby, G., Brumar, C. D., Chang, R., & Suh, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326904 · Cited by 35 · see citing works
User-Centric Explainability in Healthcare: A Knowledge-Level Perspective of Informed Machine Learning
Oberste, L., & Heinzl, A. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3227225 · Cited by 35 · see citing works
A Roadmap of Explainable Artificial Intelligence: Explain to Whom, When, What and How?
Wang, Z., Huang, C., & Yao, X. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3702004 · Cited by 32 · see citing works
Toward Practical Usage of the Attention Mechanism as a Tool for Interpretability
Tutek, M., & Šnajder, J. (2022) · IEEE Access · doi:10.1109/access.2022.3169772 · Cited by 33 · see citing works
Whose Knowledge is Valued? Epistemic Injustice in CSCW Applications
Ajmani, L. H., Foriest, J. C., Taylor, J., Pittman, K., Gilbert, S., & DeVito, M. A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687062 · Cited by 30 · see citing works
Let's Talk Futures: A Literature Review of HCI's Future Orientation
Sanchez, C., Wang, S., Savolainen, K., Epp, F. A., & Salovaara, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713759 · Cited by 30 · see citing works
Using human-in-the-loop and explainable AI to envisage new future work practices
Tsiakas, K., & Murray-Rust, D. (2022) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3529190.3534779 · Cited by 28 · see citing works
Toward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 31 · see citing works
Questioning the ability of feature-based explanations to empower non-experts in robo-advised financial decision-making
Bertrand, A., Eagan, J., & Maxwell, W. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594053 · Cited by 27 · see citing works
How can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?
Balayn, A., Rikalo, N., Lofi, C., Yang, J., & Bozzon, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517474 · Cited by 24 · see citing works
How to Choose an Explainability Method? Towards a Methodical Implementation of XAI in Practice
Vermeire, T., Laugel, T., Renard, X., Martens, D., & Detyniecki, M. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-93736-2_39 · Cited by 23 · see citing works
A Human–AI interaction paradigm and its application to rhinocytology
Desolda, G., Dimauro, G., Esposito, A., Lanzilotti, R., Matera, M., & Zancanaro, M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102933 · Cited by 23 · see citing works
"If it is easy to understand then it will have value": Examining Perceptions of Explainable AI with Community Health Workers in Rural India
Okolo, C. T., Agarwal, D., Dell, N. L., & Vashistha, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637348 · Cited by 21 · see citing works
Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts
Suh, A., Appleby, G., Anderson, E. W., Finelli, L. A., Chang, R., & Cashman, D. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3259341 · Cited by 20 · see citing works
Rideshare Transparency: Translating Gig Worker Insights on AI Platform Design to Policy
Rao, V. N., Dalal, S., Agarwal, E., Calacci, D., & Monroy-Hernández, A. (2024) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3711059 · Cited by 30 · see citing works
GAM Coach: Towards Interactive and User-centered Algorithmic Recourse
Wang, Z. J., Vaughan, J. W., Caruana, R., & Chau, D. H. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580816 · Cited by 20 · see citing works
Can Requirements Engineering Support Explainable Artificial Intelligence? Towards a User-Centric Approach for Explainability Requirements
Habiba, U.-E., Bogner, J., & Wagner, S. (2022) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew56159.2022.00038 · Cited by 19 · see citing works
Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI
Donoso-Guzmán, I., Ooge, J., Parra, D., & Verbert, K. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_10 · Cited by 19 · see citing works
How do ML practitioners perceive explainability? an interview study of practices and challenges
Habiba, U.-E., Habib, M. K., Bogner, J., Fritzsch, J., & Wagner, S. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10565-2 · Cited by 19 · see citing works
Unpacking Trust Dynamics in the LLM Supply Chain: An Empirical Exploration to Foster Trustworthy LLM Production & Use
Balayn, A., Yurrita, M., Rancourt, F., Casati, F., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713787 · Cited by 22 · see citing works
Improving Governance Outcomes Through AI Documentation: Bridging Theory and Practice
Winecoff, A., & Bogen, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713814 · Cited by 16 · see citing works
Beyond Visual Analytics: Human-Machine Teaming for AI-Driven Data Sensemaking
Wenskovitch, J. E., Fallon, C. K., Miller, K. H., & Dasgupta, A. (2021) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex53765.2021.00012 · Cited by 15 · see citing works
Saliency Cards: A Framework to Characterize and Compare Saliency Methods
Boggust, A., Suresh, H., Strobelt, H., Guttag, J. V., & Satyanarayan, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593997 · Cited by 15 · see citing works
Self-citation
Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank
Sipos, L., Schäfer, U., Glinka, K., & Müller-Birn, C. (2023) · Mensch und Computer (MuC) · doi:10.1145/3603555.3608551 · Cited by 12 · see citing works
Systemization of Knowledge (SoK): Creating a Research Agenda for Human-Centered Real-Time Risk Detection on Social Media Platforms
Alsoubai, A., Park, J., Qadir, S., Stringhini, G., Razi, A., & Wiśniewski, P. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642315 · Cited by 12 · see citing works
Secondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency
Ajmani, L., Abdelkadir, N. A., & Chancellor, S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732071 · Cited by 11 · see citing works
(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court
Jin, A., & Salehi, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641902 · Cited by 10 · see citing works
Towards Visual Analytics for Explainable AI in Industrial Applications
Kucher, K., Zohrevandi, E., & Westin, C. (2025) · Analytics · doi:10.3390/analytics4010007 · Cited by 10 · see citing works
Faulty or Ready? Handling Failures in Deep-Learning Computer Vision Models until Deployment: A Study of Practices, Challenges, and Needs
Balayn, A., Rikalo, N., Yang, J., & Bozzon, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581555 · Cited by 10 · see citing works
Kaleidoscope: Semantically-grounded, context-specific ML model evaluation
Suresh, H., Shanmugam, D., Chen, T., Bryan, A. G., D’Amour, A., Guttag, J. V., & Satyanarayan, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581482 · Cited by 11 · see citing works
Self-citation
Minimum levels of interpretability for artificial moral agents
Vijayaraghavan, A., & Badea, C. (2024) · AI and Ethics · doi:10.1007/s43681-024-00536-0 · Cited by 9 · see citing works
AI is Entering Regulated Territory: Understanding the Supervisors' Perspective for Model Justifiability in Financial Crime Detection
Bertrand, A., Eagan, J., Maxwell, W., & Brand, J. L. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642326 · Cited by 7 · see citing works
Enhancing Transparency through Explainable Artificial Intelligence: An Exploratory Analysis on Collusion and Corruption Scenario in Digital Government
Sampaio, I. G. B., Fontes, S., Andrade, E. D. O., Bernardini, F., & Viterbo, J. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657274 · Cited by 7 · see citing works
Ethical and epistemic implications of artificial intelligence in medicine: a stakeholder-based assessment
Adams, J. (2025) · AI & Society · doi:10.1007/s00146-025-02398-4 · Cited by 11 · see citing works
Aspects and Views on Responsible Artificial Intelligence
Brumen, B., Göllner, S., & Tropmann-Frick, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25599-1_29 · Cited by 7 · see citing works
FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
Barker, M., Kallina, E., Ashok, D., Collins, K. M., Casovan, A., Weller, A., Talwalkar, A., Chen, V., & Bhatt, U. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623239 · Cited by 7 · see citing works
Developing science teachers’ enacted pedagogical content knowledge through integration of student feedback into the Refined Consensus Model of pedagogical content knowledge
Hartmuth, D., Amin, R. M., Schlichting, D., Schött, S., Haudenschild, J., Küchenhoff, H., & Neuhaus, B. J. (2025) · Cogent Education · doi:10.1080/2331186x.2025.2470563 · Cited by 7 · see citing works
FLAN: feature-wise latent additive neural models for biological applications
Nguyen, A.-P., Vasilaki, S., & Martínez, M. R. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad056 · Cited by 6 · see citing works
From Discovery to Adoption: Understanding the ML Practitioners’ Interpretability Journey
Ashtari, N., Mullins, R. S., Qian, C., Wexler, J., Tenney, I., & Pushkarna, M. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596046 · Cited by 6 · see citing works
Perception-Centric Explainable AI: Bridging Cognitive Theories and HCI Design for Enhanced User Experience
Alhasan, S., & Alnanih, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.03.040 · Cited by 5 · see citing works
Enhancing AI Explainability Through the EXACT Framework: A User-Centric Approach
Alhasan, S., & Alnanih, R. A. (2025) · IEEE Access · doi:10.1109/access.2025.3576234 · Cited by 5 · see citing works
Introducing the ethical-epistemic matrix: a principle-based tool for evaluating artificial intelligence in medicine
Adams, J. (2024) · AI and Ethics · doi:10.1007/s43681-024-00597-1 · Cited by 5 · see citing works
Visually Explaining Uncertain Price Predictions in Agrifood: A User-Centred Case-Study
Ooge, J., & Verbert, K. (2022) · Agriculture · doi:10.3390/agriculture12071024 · Cited by 6 · see citing works
Tensions Between the Proxies of Human Values in AI
Datta, T., Nissani, D., Cembalest, M., Khanna, A., Massa, H., & Dickerson, J. P. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00049 · Cited by 6 · see citing works
The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR
State, L., Colmenarejo, A. B., Beretta, A., Ruggieri, S., Turini, F., & Law, S. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09430-w · Cited by 6 · see citing works
Can AI Model the Complexities of Human Moral Decision-making? A Qualitative Study of Kidney Allocation Decisions
Keswani, V., Conitzer, V., Sinnott-Armstrong, W. P., Nguyen, B. K., Heidari, H., & Borg, J. S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714167 · Cited by 4 · see citing works
Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
Lee, M. H., & Tok, M. Z. Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732155 · Cited by 5 · see citing works
Explainable AI for Daily Scenarios from End-Users’ Perspective: Non-Use, Concerns, and Ideal Design
Wang, L., Anyi, C. L., Xu, K., Liu, Y., Arriaga, R. I., & Goel, A. K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735796 · Cited by 4 · see citing works
Modular oversight methodology: a framework to aid ethical alignment of algorithmic creations
Kyriakou, K., & Otterbacher, J. (2024) · Design Science · doi:10.1017/dsj.2024.23 · Cited by 4 · see citing works
A Study of Explainability Features to Scrutinize Faceted Filtering Results
Qu, J., Arguello, J., & Wang, Y. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482409 · Cited by 5 · see citing works
Communicating Performance of Regression Models Using Visualization in Pharmacovigilance
Suh, A., Appleby, G., Anderson, E. W., Finelli, L. A., & Cashman, D. (2021) · Workshop on Visual Analytics in Healthcare (VAHC) · doi:10.1109/vahc53616.2021.00006 · Cited by 4 · see citing works
Requirements Engineering for Explainable AI
Umm-e-Habiba (2023) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re57278.2023.00058 · Cited by 4 · see citing works
Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships
Boggust, A., Bang, H., Strobelt, H., & Satyanarayan, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713406 · Cited by 6 · see citing works
Self-citation
Exploring Subjective Notions of Explainability through Counterfactual Visualization of Sentiment Analysis
Crisan, A., Butters, N., & Zoe (2024) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv64461.2024.00007 · Cited by 3 · see citing works
f-RecX: A framework for designing effective textual explanations in recommender systems’ user interfaces
Al-Hazwani, I., Morgenshtern, G., El‐Assady, M., & Bernard, J. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103627 · Cited by 3 · see citing works
Explainable AI for Hate Speech Moderation: A Stakeholder‐Centered and Sociotechnical Review
Qureshi, M. D. M., Qureshi, M. A., & Rashwan, W. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70076 · Cited by 6 · see citing works
VIME: Visual Interactive Model Explorer for Identifying Capabilities and Limitations of Machine Learning Models for Sequential Decision-Making
Antar, A. D., Molaei, S., Chen, Y.-Y., Lee, M. L., & Banovic, N. (2024) · ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676323 · Cited by 8 · see citing works
ABC-GAN: Spatially Constrained Counterfactual Generation for Image Classification Explanations
Mindlin, D., Schilling, M., & Cimiano, P. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44064-9_15 · Cited by 3 · see citing works
Explainable AI For Financial and Operational Decision Support: Improving Transparency and Managerial Trust
Chy, M. T. I. (2024) · American Journal of Advanced Technology and Engineering Solutions · doi:10.63125/kwfvzw68 · Cited by 12 · see citing works
Human-AI Interfaces are a Central Component of Trustworthy AI
Plass, M., Kargl, M., Evans, T., Brčić, L., Regitnig, P., Geißler, C., Carvalho, R., Jansen, C., Zerbe, N., Holzinger, A., et al. (2022) · Intelligent systems reference library · doi:10.1007/978-3-031-12807-3_11 · Cited by 2 · see citing works
Persona Adaptable Strategies Make Large Language Models Tractable
Hattab, G., Anžel, A., Dubey, A., Ezekannagha, C., Yang, Z., & İlgen, B. (2024) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3711542.3711581 · Cited by 2 · see citing works
Human-Centered Explainable AI (XAI)
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 314 · see citing works
Toward User-centered Explainable Displays for Complex Machine Learning Models in Healthcare: A Case Study of Heart Disease Prediction
Nuamah, J. K. (2024) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813241264252 · Cited by 1 · see citing works
"Good" XAI Design: For What? In Which Ways?
Wang, L., Liu, Y., & Goel, A. K. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720036 · Cited by 1 · see citing works
Disentangling Stakeholder Role and Expertise in User-Centered Explainable AI
Szymanski, M., Abeele, V. V., & Verbert, K. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728351 · Cited by 1 · see citing works
Biasing Rule-Based Explanations Towards User Preferences
Mahya, P., & Fürnkranz, J. (2025) · Information · doi:10.3390/info16070535 · Cited by 1 · see citing works
Towards Explainable AI: A Framework for Interpretable Deep Learning in High-Stakes Domains
Manga, I. (2025) · International Conference on Soft Computing for Security Applications (ICSCSA) · doi:10.1109/icscsa66339.2025.11170778 · Cited by 1 · see citing works
Domain Experience and Expertise in Explainable AI Applications: A Bearing Fault Diagnosis Case Study
Zhao, Z., Castelle, M., & Turkay, Ç. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757583 · Cited by 1 · see citing works
SQeX: a scenario-based and question-driven framework for designing explanatory interfaces in autonomous vehicles
Zhang, Y., Wang, W., Li, J., Guo, W., & Sun, X. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2626745 · Cited by 1 · see citing works
Understanding Movement Disorder Specialists’ Workflow for Designing Digital Health Tools
Ikhile, J. A., & Pandey, V. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3803795 · Cited by 1 · see citing works
XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflows
Wang, X., Yin, M., Koh, E., & Dogan, M. D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798531 · Cited by 1 · see citing works
Speeding Things Up. Can Explainability Improve Human Learning?
Mannmeusel, J., Rothfelder, M., & Khoshrou, S. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44064-9_4 · Cited by 2 · see citing works
Towards Trustworthy and Human-Centred AI Explanations
Manresa-Yee, C., Ramis, S., Gaya-Morey, F. X., & Rubio, J. M. B. (2024) · Springer eBooks · doi:10.1007/978-3-031-71697-3_4 · Cited by 1 · see citing works
Trustworthiness Evaluation of Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_12 · Cited by 2 · see citing works

Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
Tomsett, R., Braines, D., Harborne, D., Preece, A. D., & Chakraborty, S. (2018) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.1806.07552 · Cited by 127 · see citing works
Stakeholders in Explainable AI
Preece, A., Harborne, D., Braines, D., Tomsett, R., & Chakraborty, S. (2018) · arXiv · doi:10.48550/arxiv.1810.00184 · Cited by 117 · see citing works
Machine Learning Explainability for External Stakeholders
Bhatt, U., Andrus, M., Weller, A., & Xiang, A. (2020) · arXiv · doi:10.48550/arxiv.2007.05408 · Cited by 41 · see citing works
The Role of Individual User Differences in Interpretable and Explainable Machine Learning SystemsPreprint
Gleaves, L. P., Schwartz, R., & Broniatowski, D. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06675 · Cited by 7 · see citing works
The Stakeholder Playbook for Explaining AI SystemsPreprint
Hoffman, R., Klein, G. L., Mueller, S. T., Jalaeian, M., & Tate, C. (2021) · doi:10.31234/osf.io/9pqez · Cited by 12 · see citing works

Authors: Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A.
Affiliations: Massachusetts Institute of Technology (US)
ORCID: Suresh, Harini
Gomez, Steven R.
Nam, Kevin K.
Satyanarayan, Arvind
Venue: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher: ACM
Published: 2021-05-06
Pages: 1-16
DOI: 10.1145/3411764.3445088
OpenAlex: W3123497755
Cited by: 117
References: 81
Citations by year: 2026: 15 · 2025: 24 · 2024: 28 · 2023: 32 · 2022: 14 · 2021: 4
FWCI: 9.01
Type: conference paper
Language: en
Open access: bronze
doi.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor, 1 d)
Locations: CHI Conference on Human Factors in Computing Systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: Air Force Research Laboratory (FA8750-19-2-1000)
NSF (National Science Foundation) (1900991)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Social Sciences › Safety Research › Ethics and Social Impacts of AI
Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords: model interpretability, stakeholder needs, stakeholder typology, black-box systems, accountability, formal knowledge, instrumental knowledge, personal knowledge, user study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks