CoChat
Sign Up
PreprintCited by 28Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISoftware Testing and Debugging Techniques

The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.

Explaining Explanations to Society

Open access
Gilpin, Leilani H., Testart, Cecilia, Fruchter, Nathaniel, Adebayo, Julius·2019·arXiv
DOI 10.48550/arxiv.1901.06560arXiv 1901.06560
Open this paper in CoChat — free accountOpen PDF

Abstract

There is a disconnect between explanatory artificial intelligence (XAI) methods and the types of explanations that are useful for and demanded by society (policy makers, government officials, etc.) Questions that experts in artificial intelligence (AI) ask opaque systems provide inside explanations, focused on debugging, reliability, and validation. These are different from those that society will ask of these systems to build trust and confidence in their decisions. Although explanatory AI systems can answer many questions that experts desire, they often don't explain why they made decisions in a way that is precise (true to the model) and understandable to humans. These outside explanations can be used to build trust, comply with regulatory and policy changes, and act as external validation. In this paper, we focus on XAI methods for deep neural networks (DNNs) because of DNNs' use in decision-making and inherent opacity. We explore the types of questions that explanatory DNN systems can answer and discuss challenges in building explanatory systems that provide outside explanations for societal requirements and benefit.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1901.06560

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a gap between technical XAI advances (which often are most useful to AI developers in processes of debugging) and the types of explanations that workers might need or want in actual use settings.

    “Yet, scholars have noted a gap between technical XAI advances (which often are most useful to AI developers in processes of debugging) and the types of explanations that workers might need or want in actual use settings (Gilpin et al., 2019).”

    From Algorithmic management in a work context (2021) · cites this paper as Gilpin et al., 2019

    The Abstract (p. 1) contrasts expert-focused explanations for debugging with explanations demanded by society, and the Introduction (p. 1) says current systems serve programmers or experts rather than end users. The paper does not specifically examine workers or what they need or want in actual use settings. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Understanding is crucial in building trust.

    “Understanding is crucial in building trust [23].”

    From Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations (2023) · cites this paper as [23]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. [1][1] Robert Andrews, Joachim Diederich, and Alan B Tickle. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-based systems , 8(6):373389, 1995. -
  2. [3][3] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016. - · arXiv
  3. [4][4] Chaofan Chen, Oscar Li, Alina Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. arXiv preprint arXiv:1806.10574 , 2018. - · arXiv
  4. [5][5] Joan Claybrook and Shaun Kildare. Autonomous vehicles: No driver. . . no regulation? Science , 361(6397):36-37, 2018. -
  5. [8][8] Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O'Brien, Stuart Schieber, James Waldo, David Weinberger, and Alexandra Wood. Accountability of AI under the law: The role of explanation. CoRR , abs/1711.01134, 2017. - · arXiv
  6. [9][9] Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189-1232, 2001. -
  7. [10][10] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An approach to evaluating interpretability of machine learning. arXiv preprint arXiv:1806.00069 , 2018. - · arXiv
  8. [14][14] Been Kim, Justin Gilmer, Fernanda Viegas, Ulfar Erlingsson, and Martin Wattenberg. Tcav: Relative concept importance testing with linear concept activation vectors. arXiv preprint arXiv:1711.11279 , 2017. - · arXiv
  9. [15][15] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in Neural Information Processing Systems , pages 1952-1960, 2014. -
  10. [16][16] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730 , 2017. - · arXiv
  11. [17][17] Susan Landau. Control use of data to protect privacy. Science , 347(6221):504-506, 2015. -
  12. [20][20] Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. Understanding variable importances in forests of randomized trees. In Advances in neural information processing systems , pages 431-439, 2013. -
  13. [25][25] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. CoRR , abs/1802.08129, 2018. - · arXiv
  14. [27][27] Cynthia Rudin. Algorithms for interpretable machine learning. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '14, pages 1519-1519, New York, NY, USA, 2014. ACM. -
  15. [28][28] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems , pages 3859-3869, 2017. -
  16. [29][29] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradientbased localization. See https://arxiv. org/abs/1610.02391 v3 , 7(8), 2016. -
  17. [30][30] Raymond Sheh and Isaac Monteath. Defining explainable ai for requirements analysis. KIKünstliche Intelligenz , pages 1-6, 2018. -
  18. [31][31] Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems , 41(3):647-665, 2014. -
  19. [32][32] Sarah Tan, Rich Caruana, Giles Hooker, and Yin Lou. Detecting bias in black-box models using transparent model distillation. arXiv preprint arXiv:1710.06169 , 2017. - · arXiv
  20. [34][34] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems , pages 33203328, 2014. -
  21. [35][35] Zhenlong Yuan, Yongqiang Lu, Zhaoguo Wang, and Yibo Xue. Droid-sec: deep learning in android malware detection. In ACMSIGCOMMComputerCommunicationReview , volume 44, pages 371-372. ACM, 2014. -
  22. [36][36] Xin Zhang, Armando Solar-Lezama, and Rishabh Singh. Interpreting neural network judgments via minimal, stable, and symbolic corrections. CoRR , abs/1802.07384, 2018. - · arXiv
  23. [38][38] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on , pages 2921-2929. IEEE, 2016. -
  24. [39][39] Jan Ruben Zilke, Eneldo Loza Mencía, and Frederik Janssen. Deepred-rule extraction from deep neural networks. In International Conference on Discovery Science , pages 457-473. Springer, 2016.

Cited by

Sorted by citations

Metadata

Authors
Gilpin, L. H., Testart, C., Fruchter, N., & Adebayo, J.
Affiliations
Moscow Institute of Thermal Technology (RU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1901.06560
arXiv
1901.06560
OpenAlex
W2914935482
Cited by
28
References
6
Citations by year
2025: 1 · 2024: 2 · 2022: 3 · 2021: 15 · 2020: 4 · 2019: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
explainable AI, deep neural network, regulatory compliance, external validation, model opacity, AI decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review