The DOI matches its Crossref record; 85 of the 89 references it cites were found in a registry or an index, and 4 are not checked yet.
Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
Open accessAbstract
To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders’ knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent HCI work has begun to pursue this direction [173,204], albeit without explicitly grounding it in metacognition; we suggest that a metacognitive lens offers new research and design directions for augmenting GenAI system usability.”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [173]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“In this case, instead of starting with a tabula rasa, perhaps an interpretability interface would be initialized with summaries of the model’s strengths and weaknesses (akin to a model card [84]).”
MethodExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 6×
““anticipate ethics-related failures before launch" [97], bias or mistake detection [89, 101, 103, 125], understand skewness and biases in input data [35], find unknown vulnerabilities and flaws [5, 38] (8/58) T3: Understand the extent of the information the model is using data entanglement [97], be…”
BackgroundInfluentialHuman-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743
Why and Where: A Characterization of Data Provenance
Buneman, P., Khanna, S., & Wang-Chiew, T. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44503-x_20 · Cited by 1,334
Research works printed but not linked: 52
- 2020. ML Interpretability for Scientific Discovery (MLI4SD) Workshop. https://sites.google.com/view/mli4sd-icml2020/home. Accessed: 2020-09-16.
- Agnar Aamodt and Enric Plaza. 1994. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI communications 7, 1 (1994), 39–59.
- 10.1109/ACCESS.2018.2870052Conference · DOI
- Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. 2019. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. arXiv:1909.03012 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1909.03012 arXiv:1909.03012. · arXiv
- Robert K Atkinson, Sharon J Derry, Alexander Renkl, and Donald Wortham. 2000. Learning from examples: Instructional principles from the worked examples research. Review of educational research 70, 2 (2000), 181–214.
- 10.1145/3377325.3377498arXiv:2001.08298 · DOI
- Chaofan Chen Oscar Li Daniel Tao Alina Barnett Cynthia Rudin and Jonathan K Su. 2019. This looks like that: deep learning for interpretable image recognition. In Advances in neural information processing systems. 8930–8941.
- Michael Chromik, Malin Eiband, Sarah Theres Völkel, and Daniel Buschek. 2019. Dark Patterns of Explainability, Transparency, and User Control for Intelligent Systems. Los Angeles (2019), 6.
- Gloria Dall’Alba and Jörgen Sandberg. 2006. Unveiling professional development: A critical review of stage models. Review of educational research 76, 3 (2006), 383–412.
- Arun Das and Paul Rad. 2020. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. arXiv:2006.11371 [cs] (June 2020). http://arxiv.org/abs/2006.11371 arXiv:2006.11371. · arXiv
- Amit Dhurandhar Vijay Iyengar Ronny Luss and Karthikeyan Shanmugam. 2017. A formal framework to characterize interpretability of procedures. (2017). arXiv:1707.03886. · arXiv
- Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608 [cs, stat] (March 2017). http://arxiv.org/abs/1702.08608 arXiv:1702.08608. · arXiv
- 10.5555/7916 · DOI
- Johanna Drucker. 2012. Humanistic theory and digital scholarship. Debates in the digital humanities 150 (2012), 85–95.
- Michael Eraut. 2010. Knowledge, working practices, and learning. In Learning through practice. Springer, 37–58.
- James Fleck. 1998. Expertise: knowledge, power and tradeability. In Exploring expertise. Springer, 143–171.
- Ruth C. Fong and Andrea Vedaldi. 2017. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97. Long Beach, California, USA. http://proceedings.mlr.press/v97/goyal19a.html arXiv:1904.07451. · arXiv
- Donna Haraway. 1988. Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist studies 14, 3 (1988), 575–599.
- Johanna Hartelius. 2008. The Rhetoric of Expertise. Ph.D. Dissertation. The University of Texas at Austin.
- 10.1080/02691728.2011.578301Publisher: · DOI
- Sam Hepenstal and David McNeish. 2020. Explainable Artificial Intelligence: What Do You Need to Know?. In Augmented Cognition. Theoretical and Technological Approaches, Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.). Springer International Publishing, Cham, 266–275.
- Jessica Hullman and Nick Diakopoulos. 2011. Visualization rhetoric: Framing effects in narrative visualization. IEEE transactions on visualization and computer graphics 17, 12(2011), 2231–2240.
- Bogdan Kulynych, David Madras, Smitha Milli, Inioluwa Deborah Raji, Angela Zhou, and Richard Zemel. 2020. Participatory Approaches to Machine Learning. International Conference on Machine Learning Workshop.
- 10.1609/hcomp.v7i1.5280 · DOI
- 10.1145/3233231 · DOI · Paper page
- Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749–760.
- George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information.Psychological review 63, 2 (1956), 81.
- 10.1007/s13347-020-00405-8arXiv:2007.04068 · DOI
- Sina Mohseni, Jeremy E. Block, and Eric D. Ragan. 2020. A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning. arXiv:1801.05075 [cs] (June 2020). http://arxiv.org/abs/1801.05075 arXiv:1801.05075. · arXiv
- Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2020. A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems. arXiv:1811.11839 [cs] (April 2020). http://arxiv.org/abs/1811.11839 arXiv:1811.11839. · arXiv
- Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2018. How do humans understand explanations from machine learning systems? an evaluation of the human-interpretability of explanation. arXiv:1802.00682 (2018). · arXiv
- Prajwal Paudyal, Junghyo Lee, Azamat Kamzin, Mohamad Soudki, Ayan Banerjee, and Sandeep KS Gupta. 2019. Learn2Sign: Explainable AI for Sign Language Learning.. In IUI Workshops.
- Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Manipulating and Measuring Model Interpretability. arXiv:1802.07810 [cs] (Nov. 2019). http://arxiv.org/abs/1802.07810 arXiv:1802.07810. · arXiv
- Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, and Supriyo Chakraborty. 2018. Stakeholders in Explainable AI. arXiv:1810.00184 [cs] (Sept. 2018). http://arxiv.org/abs/1810.00184 arXiv:1810.00184. · arXiv
- Alexander Renkl. 2014. Toward an instructionally oriented theory of example-based learning. Cognitive science 38, 1 (2014), 1–37.
- Marco Tulio Ribeiro Sameer Singh and Carlos Guestrin. 2016. Model-agnostic interpretability of machine learning. (2016). arXiv:1606.05386. · arXiv
- Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI.. In IUI Workshops.
- Wojciech Samek Thomas Wiegand and Klaus-Robert Müller. 2017. Explainable artificial intelligence: Understanding visualizing and interpreting deep learning models. (2017). arXiv:1708.08296. · arXiv
- Udo Schlegel, Hiba Arnout, Mennatallah El-Assady, Daniela Oelke, and Daniel A. Keim. 2019. Towards a Rigorous Evaluation of XAI Methods on Time Series. arXiv:1909.07082 [cs] (Sept. 2019). http://arxiv.org/abs/1909.07082 arXiv:1909.07082. · arXiv
- Johannes Schneider and Joshua Handali. 2019. PERSONALIZED EXPLANATION FOR MACHINE LEARNING: A CONCEPTUALIZATION. In Proceedings of the 27th European Conference on Information Systems (ECIS). Stockholm & Uppsala, Sweden. https://aisel.aisnet.org/ecis2019_rp/171
- 10.1109/TVCG.2019.2934629Conference · DOI
- Clay Spinuzzi. 2005. The methodology of participatory design. Technical communication 52, 2 (2005), 163–174.
- Mukund Sundararajan, Jinhua Xu, Ankur Taly, Rory Sayres, and Amir Najmi. 2019. Exploring Principled Visualizations for Deep Network Attributions.. In IUI Workshops, Vol. 4.
- Richard Tomsett Dave Braines Dan Harborne Alun Preece and Supriyo Chakraborty. 2018. Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems. In arXiv:1806.07552 [cs]. http://arxiv.org/abs/1806.07552 arXiv:1806.07552. · arXiv
- Ryan Weber. 2012. Review of The Rhetoric of Expertise. Rhetoric and Public Affairs 15, 1 (2012), 193–196. https://www.jstor.org/stable/41955617 Publisher: Michigan State University Press.
- Adrian Weller. 2019. Transparency: Motivations and Challenges. arXiv:1708.01870 [cs] (Aug. 2019). http://arxiv.org/abs/1708.01870 arXiv:1708.01870. · arXiv
- Jacob O Wobbrock and Julie A Kientz. 2016. Research contributions in human-computer interaction. interactions 23, 3 (2016), 38–44.
- Yao Xie Ge Gao and Xiang ’Anthony’ Chen. 2019. Outlining the Design Space of Explainable Intelligent Systems for Medical Diagnosis. arxiv:1902.06019 [cs.HC] · arXiv
- Jill Yielder. 2001. Professional Expertise: A Model for Integration and Change. Thesis. ResearchSpace@Auckland. https://researchspace.auckland.ac.nz/handle/2292/2340Accepted: 2008-01-30T02:04:57Z.
- 10.1007/978-3-319-10590-1_53Series · DOI
- Ruijing Zhao, Izak Benbasat, and Hasan Cavusoglu. 2019. Transparency in Advice-Giving Systems: A Framework and a Research Model for Transparency Provision.. In IUI Workshops.
Cited by
Sorted by citations
Items 101–119 of 119
Tell Me Without Telling Me: Two-Way Prediction of Visualization Literacy and Visual Attention
Chang, M., Wang, Y., Wang, H. W., Zhou, Y., Bulling, A., & Xiong, C. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3634815 · Cited by 5
Gupta, T. (2023) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac58438.2023.10290593
Yang, N.-E., Han, E.-B., & Choi, J. (2024) · Journal of Digital Contents Society · doi:10.9728/dcs.2024.25.8.2261
Raymond, C., Ratté, S., & Daoust, M. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10837695
Zhang, L., & Scifleet, P. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720072
Tool-supported Development of ML Prototypes
Aydın, S., & Lichter, H. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00063
Bertrand, A. (2024) · theses.fr (ABES) · doi:10.70675/216d3647z78bcz4c68z9ba9z2d0a029cf1b4 · Cited by 1
XEQ Scale for Evaluating XAI Experience Quality
Wijekoon, A., Wiratunga, N., Corsar, D., Martin, K., Nkisi-Orji, I., Pirie, C., Díaz-Agudo, B., & Bridge, D. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3820897
From Curiosity to Caution: How Expertise Shapes the Use of Interpretable Machine Learning
Naqvi, S., Balaji, G., Laju, A., & Kaur, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812376
Preaching Meaning, Practicing the Motions: A Critical Analysis of XAI Research
van der Linden, I., & Ghebreab, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812356
On the Interplay of Explainability and Fairness in AI: A Survey
Fragkathoulas, C., Papanikou, V., Karidi, D. P., & Pitoura, E. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3706635
Kladko, S. (2026) · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit70063.2026.11580622
Tao, J., & Zhou, L. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10815-2
Dafali, S. M., Kissi, M., & Beggar, O. E. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.112856 · Cited by 1
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.010
Law, E., & Yin, M. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.008
Doing Human-Centered Machine Learning
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.006
A Study on a Margin-Based Adaptive UX Framework for Human–Agentic AI Collaboration
Ahn, S. (2026) · Journal of Broadcast Engineering · doi:10.5909/jbe.2026.31.5.948
Metadata
- Authors
- Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-16
- OpenAlex
- W3123497755
- Cited by
- 117
- References
- 81
- Citations by year
- 2026: 15 · 2025: 24 · 2024: 28 · 2023: 32 · 2022: 14 · 2021: 4
- FWCI
- 9.01
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Air Force Research Laboratory (FA8750-19-2-1000)
- NSF (National Science Foundation) (1900991)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- model interpretability, stakeholder needs, stakeholder typology, black-box systems, accountability, formal knowledge, instrumental knowledge, personal knowledge, user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
