CoChat
Sign Up
Cited by 119Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISoftware Testing and Debugging Techniques

The DOI matches its Crossref record; 85 of the 89 references it cites were found in a registry or an index, and 4 are not checked yet.

Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Open access
Suresh, Harini, Gomez, Steven R., Nam, Kevin K., Satyanarayan, Arvind·2021·Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3411764.3445088
Open this paper in CoChat — free accountOpen PDF

Abstract

To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders’ knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3411764.3445088

Cites

85 of 89 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 52

  1. 2020. ML Interpretability for Scientific Discovery (MLI4SD) Workshop. https://sites.google.com/view/mli4sd-icml2020/home. Accessed: 2020-09-16.
  2. Agnar Aamodt and Enric Plaza. 1994. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI communications 7, 1 (1994), 39–59.
  3. 10.1109/ACCESS.2018.2870052Conference · DOI
  4. Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. 2019. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. arXiv:1909.03012 [cs, stat] (Sept. 2019). http://arxiv.org/abs/1909.03012 arXiv:1909.03012. · arXiv
  5. Robert K Atkinson, Sharon J Derry, Alexander Renkl, and Donald Wortham. 2000. Learning from examples: Instructional principles from the worked examples research. Review of educational research 70, 2 (2000), 181–214.
  6. 10.1145/3377325.3377498arXiv:2001.08298 · DOI
  7. Chaofan Chen Oscar Li Daniel Tao Alina Barnett Cynthia Rudin and Jonathan K Su. 2019. This looks like that: deep learning for interpretable image recognition. In Advances in neural information processing systems. 8930–8941.
  8. Michael Chromik, Malin Eiband, Sarah Theres Völkel, and Daniel Buschek. 2019. Dark Patterns of Explainability, Transparency, and User Control for Intelligent Systems. Los Angeles (2019), 6.
  9. Gloria Dall’Alba and Jörgen Sandberg. 2006. Unveiling professional development: A critical review of stage models. Review of educational research 76, 3 (2006), 383–412.
  10. Arun Das and Paul Rad. 2020. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. arXiv:2006.11371 [cs] (June 2020). http://arxiv.org/abs/2006.11371 arXiv:2006.11371. · arXiv
  11. Amit Dhurandhar Vijay Iyengar Ronny Luss and Karthikeyan Shanmugam. 2017. A formal framework to characterize interpretability of procedures. (2017). arXiv:1707.03886. · arXiv
  12. Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608 [cs, stat] (March 2017). http://arxiv.org/abs/1702.08608 arXiv:1702.08608. · arXiv
  13. 10.5555/7916 · DOI
  14. Johanna Drucker. 2012. Humanistic theory and digital scholarship. Debates in the digital humanities 150 (2012), 85–95.
  15. Michael Eraut. 2010. Knowledge, working practices, and learning. In Learning through practice. Springer, 37–58.
  16. James Fleck. 1998. Expertise: knowledge, power and tradeability. In Exploring expertise. Springer, 143–171.
  17. Ruth C. Fong and Andrea Vedaldi. 2017. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
  18. Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97. Long Beach, California, USA. http://proceedings.mlr.press/v97/goyal19a.html arXiv:1904.07451. · arXiv
  19. Donna Haraway. 1988. Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist studies 14, 3 (1988), 575–599.
  20. Johanna Hartelius. 2008. The Rhetoric of Expertise. Ph.D. Dissertation. The University of Texas at Austin.
  21. 10.1080/02691728.2011.578301Publisher: · DOI
  22. Sam Hepenstal and David McNeish. 2020. Explainable Artificial Intelligence: What Do You Need to Know?. In Augmented Cognition. Theoretical and Technological Approaches, Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.). Springer International Publishing, Cham, 266–275.
  23. Jessica Hullman and Nick Diakopoulos. 2011. Visualization rhetoric: Framing effects in narrative visualization. IEEE transactions on visualization and computer graphics 17, 12(2011), 2231–2240.
  24. Bogdan Kulynych, David Madras, Smitha Milli, Inioluwa Deborah Raji, Angela Zhou, and Richard Zemel. 2020. Participatory Approaches to Machine Learning. International Conference on Machine Learning Workshop.
  25. 10.1609/hcomp.v7i1.5280 · DOI
  26. 10.1145/3233231 · DOI · Paper page
  27. Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749–760.
  28. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information.Psychological review 63, 2 (1956), 81.
  29. 10.1007/s13347-020-00405-8arXiv:2007.04068 · DOI
  30. Sina Mohseni, Jeremy E. Block, and Eric D. Ragan. 2020. A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning. arXiv:1801.05075 [cs] (June 2020). http://arxiv.org/abs/1801.05075 arXiv:1801.05075. · arXiv
  31. Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2020. A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems. arXiv:1811.11839 [cs] (April 2020). http://arxiv.org/abs/1811.11839 arXiv:1811.11839. · arXiv
  32. Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2018. How do humans understand explanations from machine learning systems? an evaluation of the human-interpretability of explanation. arXiv:1802.00682 (2018). · arXiv
  33. Prajwal Paudyal, Junghyo Lee, Azamat Kamzin, Mohamad Soudki, Ayan Banerjee, and Sandeep KS Gupta. 2019. Learn2Sign: Explainable AI for Sign Language Learning.. In IUI Workshops.
  34. Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Manipulating and Measuring Model Interpretability. arXiv:1802.07810 [cs] (Nov. 2019). http://arxiv.org/abs/1802.07810 arXiv:1802.07810. · arXiv
  35. Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, and Supriyo Chakraborty. 2018. Stakeholders in Explainable AI. arXiv:1810.00184 [cs] (Sept. 2018). http://arxiv.org/abs/1810.00184 arXiv:1810.00184. · arXiv
  36. Alexander Renkl. 2014. Toward an instructionally oriented theory of example-based learning. Cognitive science 38, 1 (2014), 1–37.
  37. Marco Tulio Ribeiro Sameer Singh and Carlos Guestrin. 2016. Model-agnostic interpretability of machine learning. (2016). arXiv:1606.05386. · arXiv
  38. Mireia Ribera and Agata Lapedriza. 2019. Can we do better explanations? A proposal of user-centered explainable AI.. In IUI Workshops.
  39. Wojciech Samek Thomas Wiegand and Klaus-Robert Müller. 2017. Explainable artificial intelligence: Understanding visualizing and interpreting deep learning models. (2017). arXiv:1708.08296. · arXiv
  40. Udo Schlegel, Hiba Arnout, Mennatallah El-Assady, Daniela Oelke, and Daniel A. Keim. 2019. Towards a Rigorous Evaluation of XAI Methods on Time Series. arXiv:1909.07082 [cs] (Sept. 2019). http://arxiv.org/abs/1909.07082 arXiv:1909.07082. · arXiv
  41. Johannes Schneider and Joshua Handali. 2019. PERSONALIZED EXPLANATION FOR MACHINE LEARNING: A CONCEPTUALIZATION. In Proceedings of the 27th European Conference on Information Systems (ECIS). Stockholm & Uppsala, Sweden. https://aisel.aisnet.org/ecis2019_rp/171
  42. 10.1109/TVCG.2019.2934629Conference · DOI
  43. Clay Spinuzzi. 2005. The methodology of participatory design. Technical communication 52, 2 (2005), 163–174.
  44. Mukund Sundararajan, Jinhua Xu, Ankur Taly, Rory Sayres, and Amir Najmi. 2019. Exploring Principled Visualizations for Deep Network Attributions.. In IUI Workshops, Vol. 4.
  45. Richard Tomsett Dave Braines Dan Harborne Alun Preece and Supriyo Chakraborty. 2018. Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems. In arXiv:1806.07552 [cs]. http://arxiv.org/abs/1806.07552 arXiv:1806.07552. · arXiv
  46. Ryan Weber. 2012. Review of The Rhetoric of Expertise. Rhetoric and Public Affairs 15, 1 (2012), 193–196. https://www.jstor.org/stable/41955617 Publisher: Michigan State University Press.
  47. Adrian Weller. 2019. Transparency: Motivations and Challenges. arXiv:1708.01870 [cs] (Aug. 2019). http://arxiv.org/abs/1708.01870 arXiv:1708.01870. · arXiv
  48. Jacob O Wobbrock and Julie A Kientz. 2016. Research contributions in human-computer interaction. interactions 23, 3 (2016), 38–44.
  49. Yao Xie Ge Gao and Xiang ’Anthony’ Chen. 2019. Outlining the Design Space of Explainable Intelligent Systems for Medical Diagnosis. arxiv:1902.06019 [cs.HC] · arXiv
  50. Jill Yielder. 2001. Professional Expertise: A Model for Integration and Change. Thesis. ResearchSpace@Auckland. https://researchspace.auckland.ac.nz/handle/2292/2340Accepted: 2008-01-30T02:04:57Z.
  51. 10.1007/978-3-319-10590-1_53Series · DOI
  52. Ruijing Zhao, Izak Benbasat, and Hasan Cavusoglu. 2019. Transparency in Advice-Giving Systems: A Framework and a Research Model for Transparency Provision.. In IUI Workshops.

Cited by

Sorted by citations

See all 119

Related

See all

Metadata

Authors
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A.
Affiliations
Massachusetts Institute of Technology (US)
ORCID
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-16
DOI
10.1145/3411764.3445088
OpenAlex
W3123497755
Cited by
117
References
81
Citations by year
2026: 15 · 2025: 24 · 2024: 28 · 2023: 32 · 2022: 14 · 2021: 4
FWCI
9.01
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor, 1 d)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Air Force Research Laboratory (FA8750-19-2-1000)
  • NSF (National Science Foundation) (1900991)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
model interpretability, stakeholder needs, stakeholder typology, black-box systems, accountability, formal knowledge, instrumental knowledge, personal knowledge, user study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review