CoChat
Sign Up
Cited by 262Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISafety Systems Engineering in Autonomy

Four principles of explainable artificial intelligence

Open access
Phillips, P Jonathon, Hahn, Carina A, Fontana, Peter C, Yates, Amy N, Greene, Kristen, Broniatowski, David A, Przybocki, Mark A·2021
DOI 10.6028/nist.ir.8312

Published version of a preprint

Open this paper in CoChat — free account

Abstract

We introduce four principles for explainable artificial intelligence (AI) that comprise fundamental properties for explainable AI systems. We propose that explainable AI systems deliver accompanying evidence or reasons for outcomes and processes; provide explanations that are understandable to individual users; provide explanations that correctly reflect the system s process for generating the output; and that a system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output. We have termed these four principles as explanation, meaningful, explanation accuracy, and knowledge limits, respectively. Through significant stakeholder engagement, these four principles were developed to encompass the multidisciplinary nature of explainable AI, including the fields of computer science, engineering, and psychology. Because one-sizefits-all explanations do not exist, different users will require different types of explanations. We present five categories of explanation and summarize theories of explainable AI. We give an overview of the algorithms in the field that cover the major classes of explainable algorithms. As a baseline comparison, we assess how well explanations provided by people follow our four principles. This assessment provides insights to the challenges of designing explainable AI systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.6028/nist.ir.8312

Cites

142 of 142 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Any explanation must address its own epistemic limits.

    “Furthermore, any explanation must address its own epistemic limits.”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [48]

    Section 2.4 says explainable AI systems should identify and declare when an answer falls outside their design or is unreliable; Section 9 likewise says the system should express its knowledge limits. The paper supports acknowledging limits in this setting, but does not require every explanation, in general, to address its own epistemic limits. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • NIST argues that AI tools should succeed or fail on the same metrics by which humans performing the same tasks succeed or fail.

    “NIST argues that AI tools should succeed or fail on the same metrics by which humans performing the same tasks succeed or fail [ 48 ].”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [48]

    The Introduction and Section 8 use human performance as a baseline for comparing explainable AI, and the Discussion applies the same explainability principles to humans and AI. But Section 8 explicitly leaves open whether machines should be held to a higher standard. The paper does not argue that AI tools should succeed or fail by the same metrics as humans across the tasks they perform. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • In most cases, as NIST explains, additional models must be reverse-engineered to trace how black box tools reached a conclusion.

    “it will be sufficient to note the necessity in most cases, as explained by NIST, of reverse-engineering additional models to trace the paths by which black box tools came to a given conclusion.”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [48]

    Section 6.2 (page not given) describes post-hoc explanations that use a model to approximate an opaque model, and Section 6 says such explanations can be generated without knowing how the original algorithm works. The paper does not say that reverse-engineering additional models is necessary in most cases or that an approximate model traces the actual path to a conclusion. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Self-interpretable models such as decision trees or rule lists let users trace the logical paths from input to output.

    “AI models reliant on decision trees or rule lists allow users to walk through the logical paths that lead a given input through the process of it becoming an output.”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [48]

  • Many current local explanation models work by varying actual data to identify which aspects of a dataset or item were decisive in an AI tool's actions.

    “many of the current models operate by providing variations of actual data submitted within a given sample to identify which aspects of a dataset or an item provided the decisive factors leading up to an AI tool's actions.”

    From Positioning Paradata: A Conceptual Frame for AI Processual Documentation in Archives and Recordkeeping Contexts (2023) · cites this paper as [48]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

Research works printed but not linked: 14

  1. [2]2. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9505-9515.
  2. [3]3. Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/8160-sanity-checks-for-saliency-maps. pdf.
  3. [6]6. David Alvarez Melis and Tommi Jaakkola. Towards Robust Interpretabil- ity with Self-Explaining Neural Networks. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems 31, pages 7775- 7784.
  4. [17]17. Charles F Bond and Bella M DePaulo. Accuracy of Deception Judgments Character- izations of Deception. Personality and Social Psychology Review, 10(3):214-234, 2006. 24 ______________________________________________________________________________________________________
  5. [36]36. Google LLC. AI Explanations Whitepaper. pages 1-28, 2019.
  6. [48]48. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ fe4b8556000d0f0cae99daa5c5c5a410-Paper.pdf.
  7. [57]57. Been Kim, Cynthia Rudin, and Julie A Shah. The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classifica- tion. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 1952-1960.
  8. [61]61. Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felton, Joel R. Reiden- berg, David G. Robinson, and Harlan Yu. Accountable Algorithms. University of Pennsylvania Law Review, pages 633-705, 2017.
  9. [84]84. Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Pre- dictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems 30, pages 4765-4774.
  10. [88]88. Smitha Milli, Ludwig Schmidt, Anca D. Dragan, and Moritz Hardt. Model recon- struction from model explanations, 2018. URL https://arxiv.org/abs/1807.05185v1. · arXiv
  11. [103]103. Rüdiger F. Pohl, editor. Cognitive illusions: A handbook on fallacies and biases in thinking, judgement and memory. Psychology Press, 2004.
  12. [123]123. Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep net- works via gradient-based localization. In Proceedings of the IEEE International 33 ______________________________________________________________________________________________________ Conference on Computer Vision, pages 618-626, 2017.
  13. [142]142. Ellen M Voorhees. System Explanations : A Cautionary Tale. In ACM CHI Work- shop on Operationalizing Human-Centered Perspectives in Explainable AI, 2021.
  14. [150]150. Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I. Inouye, and Pradeep K. Ravikumar. On the (In)fidelity and Sensitivity of Expla- nations. pages 10967-10978, 2019. URL http://papers.neurips.cc/paper/ 9278-on-the-infidelity-and-sensitivity-of-explanations.

Cited by

Sorted by citations

Related

Authors
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A.
Affiliations
National Institute of Standards and Technology (US)
ORCID
Publisher
National Institute of Standards and Technology (U.S.)
Published
2021-09-29
DOI
10.6028/nist.ir.8312
OpenAlex
W3204153914
Cited by
262
References
142
Citations by year
2026: 63 · 2025: 67 · 2024: 67 · 2023: 34 · 2022: 20 · 2021: 11
Type
report
Language
en
Open access
Funders
  • National Institute of Standards and Technology
  • Information Technology Laboratory
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
explainable AI, stakeholder engagement, human explanations, trust in AI outputs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review