CoChat
Sign Up
PreprintCited by 2

The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.

Principles of Explanation in Human-AI Systems

Mueller, Shane T., Veinott, Elizabeth S., Hoffman, Robert R., Klein, Gary, Alam, Lamia, Mamun, Tauseef, Clancey, William J.·2021·arXiv
DOI 10.48550/arxiv.2102.04972arXiv 2102.04972
Open this paper in CoChat — free account

Abstract

Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea about explainability. These systems are often not tested to determine whether the algorithm helps users accomplish any goals, and so their explainability remains unproven. We propose an alternative: to start with human-focused principles for the design, testing, and implementation of XAI systems, and implement algorithms to serve that purpose. In this paper, we review some of the basic concepts that have been used for user-centered XAI systems over the past 40 years of research. Based on these, we describe the "Self-Explanation Scorecard", which can help developers understand how they can empower users by enabling self-explanation. Finally, we present a set of empirically-grounded, user-centered design principles that may guide developers to create successful explainable systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.04972

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Common human-centric evaluation criteria used in the context of XAI are related to understanding, user's mental model, trust and reliance, satisfaction, usefulness and usability, performance, fairness.

    “Common human-centric evaluation criteria that are used in the context of XAI are related to understanding, user's mental model, trust and reliance, satisfaction, usefulness and usability, performance, fairness (Mueller et al., 2021;Mohseni et al., 2021;Lai, Chen, Liao, Smith-Renner, & Tan, 2021).”

    From Trust, distrust, and appropriate reliance in (X)AI: A conceptual clarification of user trust and survey of its empirical evaluation (2025) · cites this paper as Mueller et al., 2021

    The Measurement Domains section (page unavailable) identifies user satisfaction, trust and reliance, mental models and understanding, and human–AI performance as ways to assess explanations. The Summary and User-Centered Explanation in AI sections discuss usefulness and usability as design goals, but our copy does not identify fairness as a human-centric evaluation criterion; it mentions fairness only as a context for algorithmic decision-making. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Alam, L. 2020. Investigating the Impact of Explanation on Re- pairing Trust in Ai Diagnostic Systems for Re-Diagnosis (Publi- cation No. 28088930) [Master's Thesis, Michigan Technological University]. ProQuest Dissertations Publishing.
  2. Aloimonos, J., Weiss, I., and Bandyopadhyay, A. 1988. Active vision. International journal of computer vision, 1(4), 333-356.
  3. Brézillon, P. 1994. Context needs in cooperative building of ex- planations. In First European Conference on Cognitive Science in Industry (pp. 443-450).
  4. Clancey, W. J. 1986. From GUIDON to NEOMYCIN and HER- ACLES in twenty short lessons. AI Magazine, 7(3), 40.
  5. Chari, S., Seneviratne, O., Gruen, D. M., Foreman, M. A., Das, A. K., & McGuinness, D. L. (2020, November). Explanation On- tology: A Model of Explanations for User-Centered AI. In Inter- national Semantic Web Conference (pp. 228-243). Springer.
  6. Conant, R. C., and Ashby, W R. 1970. Every good regulator of a system must be a model of that system. International Journal of Systems Science, 1(2), 89-97.
  7. Deal, S.V. and Hoffman, R.R. (2010, September/October). The Practitioner's Cycles Part 3: Implementation problems. IEEE In- telligent Systems, pp. 77-81.
  8. Doyle, J. K., and Ford, D., 1998, Mental models concepts for sys- tem dynamics research, System Dynamics Review, 14(1), 3-29.
  9. Doyle, D., Tsymbal, A., & Cunningham, P. 2003. A review of ex- planation and explanation in case-based reasoning. Trinity Col- lege Dublin, Department of Computer Science Technical Report TCD-CS-2003-41. http://www.tara.tcd.ie/handle/2262/12919
  10. Fallon, C. K., and Blaha, L. M. 2018. Improving Automation Transparency: Addressing Some of Machine Learning's Unique Challenges. International Conference on Augmented Cognition, 245-254. Springer.
  11. Forrester, J. 1961. Industrial Dynamics. Cambridge, MA: Pro- ductivity Press
  12. Findlay, J. M., Findlay, J. M., and Gilchrist, I. D. 2003. Active vi- sion: The psychology of looking and seeing (No. 37). Oxford University Press.
  13. Greenbaum, J. & Kyng, M. (Eds.). 1991. Design at work: Coop- erative design of computer systems. Hillsdale, NJ: Lawrence Erl- baum Associates.
  14. Guerlain, S. 1995. Using the critiquing approach to cope with brittle expert systems. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 39, pp. 233-237). SAGE Publications Sage CA: Los Angeles, CA.
  15. Greenbaum, J. and Kyng, M. (Eds.) 1991. Design at work: Coop- erative design of computer systems. Hillsdale, NJ: Lawrence Erl- baum Associates.
  16. Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., and Darrell, T. 2016. Generating visual explanations. Euro- pean Conference on Computer Vision, 3-19. http://link.springer.- com/chapter/10.1007/978-3-319-46493-0_1 · DOI · Paper page
  17. Hoffman, R. R., Deal, S. V., Potter, S., and Roth, E. M. 2010. The practitioner's cycles, part 2: Solving envisioned world problems. IEEE Intelligent Systems, 25(3), 6-11.
  18. Hoffman, R.R. (ed) 2012. Collected Essays on Human-Centered Computing, 2001-2011. New York: IEEE Computer Soc. Press.
  19. Hoffman, R. R., Klein, G., and Mueller, S. T. 2018b. Explaining explanation for "Explainable AI". Proceedings of the 2018 con- ference of the Human Factors and Ergonomic Society (HFES), Philadelphia PA, October 2018.
  20. Hoffman, R. R., Mueller, S. T., and Klein, G. 2017. Explaining Explanation, Part 2: Empirical Foundations. IEEE Intelligent Sys- tems, 32(4), 78-86.
  21. Hoffman, R., Miller, T., Mueller, S. T., Klein, G. and Clancey, W. J. 2018c. Explaining Explanation, Part 4: A Deep Dive on Deep Nets. IEEE Intelligent Systems, 33(3), 87-95.
  22. Hoffman, R. R., Mueller, S. T., Klein, G., and Litman, J. 2018a. Metrics for explainable AI: Challenges and prospects. arXiv:1812.04608. · arXiv
  23. Hoffman, R. R., Clancey, W. J., and Mueller, S. T. 2020. Ex- plaining AI as an Exploratory Process: The Peircean Abduction Model. arXiv:2009.14795. · arXiv
  24. Karat, C. M. 1998. Guaranteeing rights for the user. Communica- tions of the ACM, 41(12), 29-31.
  25. Karsenty, L., and Brezillon, P. J. 1995. Cooperative problem solving and explanation. Expert Systems with Applications, 8(4), 445-462.
  26. Kass, R., and Finin, T. 1988. The need for user models in generat- ing expert system explanation. International Journal of Expert Systems, 1(4), 345-375.
  27. Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. 2020. Interpreting Interpretability: Under- standing Data Scientists' Use of Interpretability Tools for Ma- chine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14).
  28. Kim, B., Khanna, R., and Koyejo, O. O. 2016. Examples are not enough, learn to criticize! Criticism for interpretability. In Ad- vances in Neural Information Processing Systems (2280-2288).
  29. Klein, G. and Hoffman, R. R. 2008. Macrocognition, mental models, and cognitive task analysis methodology. In J. M. Schraagen, L. G. Militello, T. Ormerod and R. Lipshitz (Eds.), Naturalistic decision making and macrocognition (pp. 57-80). Aldershot, England: Ashgate.
  30. Kulesza, T., Burnett, M., Wong, W.-K., and Stumpf, S. 2015. Principles of explanatory debugging to personalize interactive machine learning. Proceedings of the 20th International Confer- ence on Intelligent User Interfaces, 126-137.
  31. Lakkaraju, H., Kamar, E., Caruana, R., and Leskovec, J. 2017. In- terpretable & explorable approximations of black box models. ArXiv Preprint ArXiv:1707.01154. · arXiv
  32. Leake, D. B. 1995. Abduction, experience, and goals: A model of everyday abductive explanation. Journal of Experimental & The- oretical Artificial Intelligence, 7(4), 407-428.
  33. Mountford, S. J. 1998. A history of the Apple human interface group. ACM SIGCHI Bulletin, 30(2), 144-146.
  34. Mueller, S. T., Agarwal, P. Linja, A, Dave, N., and Alam, L. 2020. The unreasonable ineptitude of deep image classification networks. Proceedings of the 64 th annual meeting of the Human Factors Society.
  35. Mueller, S.T., Hoffman, R.R., Clancey, W, Emrey, A., and Klein, G. 2019. "Explanation in Human-AI Systems: A Literature Meta- Review, Synopsis of Key Ideas and Publications, and Bibliogra- phy for Explainable AI." Report on Award No. FA8650-17-2- 7711, DARPA XAI Program. DTIC AD1073994.
  36. Mueller, S.T. and Klein, G. 2011. Improving Users' Mental Mod- els of Intelligent Software Tools," Intelligent Systems, IEEE, 26(2), 77-83.
  37. Naiseh, M., Jiang, N., Ma, J., & Ali, R. 2020. Personalising Ex- plainable Recommendations: Literature and Conceptualisation. In World Conference on Information Systems and Technologies (518-533). Springer, Cham.
  38. Ribeiro, M. T., Singh, S., and Guestrin, C. 2016. Why Should I Trust You?: Explaining the Predictions of Any Classifier. In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135-1144). ACM.
  39. Rozenblit, L., and Keil, F. 2002. The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive science, 26(5), 521-562.
  40. Rouse, W. B., and Morris, N. M. 1986. On looking into the black box: Prospects and limits in the search for mental models. Psy- chological Bulletin, 100(3), 349.
  41. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. 2017. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618- 626).
  42. Swartout, W. R. 1977. A digitalis therapy advisor with explana- tions. Proceedings of the 5th International Joint Conference on Artificial Intelligence-Volume 2, 819-825. http://dl.acm.org/cita- tion.cfm?id=1623009
  43. Tate, D., Grier, R., A. Martin, C., L. Moses, F., and Sparrow, D. 2016. A Framework for evidence based licensure of adaptive au- tonomous systems. IDA Paper P-5325. 10.13140/ RG.2.2.11845.86247 · DOI
  44. Veinott, E., Klein, G., and Wiggins, S. 2010. Evaluating the ef- fect of the Premortem Method on Plan confidence. Paper pre- sented at the 7th International ISCRAM Conference 2010.
  45. Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. 2019. Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-15).110111
  46. Wick, M. R., and Thompson, W. B. 1992. Reconstructive expert system explanation. Artificial Intelligence, 54(1-2), 33-70.
  47. Yang, S. C. H., and Shafto, P. 2017. Explainable artificial intelli- gence via Bayesian Teaching. In NIPS 2017 workshop on Teach- ing Machines, Robots, and Humans (pp. 127-137).
  48. Zeiler, M. D., and Fergus, R. 2014. Visualizing and understand- ing convolutional networks. In European conference on computer vision (pp. 818-833). Springer, Cham.

Cited by

Sorted by citations

Metadata

Authors
Mueller, S. T., Veinott, E. S., Hoffman, R. R., Klein, G., Alam, L., Mamun, T., & Clancey, W. J.
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2102.04972
arXiv
2102.04972
References
9
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review