CoChat
Sign Up
PreprintCited by 218Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIUsability and User Interface Design

Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences

Open access
Miller, Tim, Howe, Piers, Sonenberg, Liz·2017·arXiv
DOI 10.48550/arxiv.1712.00547arXiv 1712.00547
Open this paper in CoChat — free accountOpen PDF

Abstract

In his seminal book `The Inmates are Running the Asylum: Why High-Tech Products Drive Us Crazy And How To Restore The Sanity' [2004, Sams Indianapolis, IN, USA], Alan Cooper argues that a major reason why software is often poorly designed (from a user perspective) is that programmers are in charge of design decisions, rather than interaction designers. As a result, programmers design software for themselves, rather than for their target audience, a phenomenon he refers to as the `inmates running the asylum'. This paper argues that explainable AI risks a similar fate. While the re-emergence of explainable AI is positive, this paper argues most of us as AI researchers are building explanatory agents for ourselves, rather than for the intended users. But explainable AI is more likely to succeed if researchers and practitioners understand, adopt, implement, and improve models from the vast and valuable bodies of research in philosophy, psychology, and cognitive science, and if evaluation of these models is focused more on people than on technology. From a light scan of literature, we demonstrate that there is considerable scope to infuse more results from the social and behavioural sciences into explainable AI, and present some key results from these fields that are relevant to explainable AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1712.00547

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current explainable models are designed to be used by ML engineers for debugging purposes, with end-users not the main target.

    “Another research [49] showed that current explainable models are designed to be used by ML engineers for debugging purposes where end-users are not the main target of these explanations.”

    From Explainable recommendation: when design meets trust calibration (2021) · cites this paper as [49]

    The Abstract (p. 1) argues that most AI researchers are building explanatory agents for themselves rather than their intended users, supporting the point that end-users are often not the main target. The paper does not say that current explainable models are designed for ML engineers to use for debugging; 'explanatory debugging' appears only in the title of a paper listed in the appendix (p. 6). (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. [Buchanan and Shortliffe, 1984] Bruce Buchanan and Edward Shortliffe. Rule-based expert systems: the MYCIN experiments of the Stanford Heuristic Programming Project . Addison-Wesley, 1984.
  2. [Chandrasekaran et al. , 1989] B Chandrasekaran, Michael C. Tanner, and John R. Josephson. Explaining control strategies in problem solving. IEEE Expert , 4(1):9-15, 1989.
  3. [Kelley, 1967] Harold H Kelley. Attribution theory in social psychology. In Nebraska symposium on motivation , pages 192-238. Uni. Nebraska Press, 1967.
  4. [Lombrozo, 2007] Tania Lombrozo. Simplicity and probability in causal explanation. Cognitive psychology , 55(3):232-257, 2007.
  5. [Lombrozo, 2012] Tania Lombrozo. Explanation and abductive inference. Oxford handbook of thinking and reasoning , pages 260-276, 2012.
  6. , [Malle, 2004] Bertram F Malle. How the mind explains behavior: Folk explanations, meaning, and social interaction . MIT Press, 2004.
  7. . [Malle, 2011] Bertram F Malle. Time to give up the dogmas of attribution: An alternative theory of behavior explanation. Advances in Experimental Social Psychology , 44(1):297-311, 2011.
  8. [Eiter and Lukasiewicz, 2002] Thomas Eiter and Thomas Lukasiewicz. Complexity results for structure-based causality. Artificial Intelligence , 142(1):53-89, 2002.
  9. [Girotto et al. , 1991] Vittorio Girotto, Paolo Legrenzi, and Antonio Rizzo. Event controllability in counterfactual thinking. Acta Psychologica , 78(1):111-133, 1991.
  10. [Grice, 1975] Herbert P Grice. Logic and conversation. In Syntax and semantics 3: Speech arts , pages 41-58. New York: Academic Press, 1975.
  11. [Hempel and Oppenheim, 1948] Carl G Hempel and Paul Oppenheim. Studies in the logic of explanation. Philosophy of Science , 15(2):135-175, 1948.
  12. [Hilton and Slugoski, 1986] Denis J Hilton and Ben R Slugoski. Knowledge-based causal attribution: The abnormal conditions focus model. Psychological review , 93(1):75, 1986.
  13. [Hilton, 1990] Denis J Hilton. Conversational processes and causal explanation. Psychological Bulletin , 107(1):65-81, 1990.
  14. [Hilton, 1996] Denis J Hilton. Mental models and causal explanation: Judgements of probable cause and explanatory relevance. Thinking & Reasoning , 2(4):273308, 1996.
  15. [Kahneman and Tversky, 1982] Daniel Kahneman and Amos Tversky. The simulation heuristic. In P. Slovic D. Kahneman and A. Tversky, editors, Judgment under Uncertainty: Heuristics and Biases . New York: Cambridge University Press, 1982.
  16. [McCloy and Byrne, 2000] Rachel McCloy and Ruth MJ Byrne. Counterfactual thinking about controllable events. Memory & Cognition , 28(6):10711078, 2000.
  17. [Miller and Gunasegaram, 1990] Dale T Miller and Saku Gunasegaram. Temporal order and the perceived mutability of events: Implications for blame assignment. Journal of personality and social psychology , 59(6):1111, 1990.
  18. [Miller, 2017] Tim Miller. Explainable AI: Insights from the social sciences. ArXiv e-prints , 1706.07269, 2017. https://arxiv.org/abs/1706.07269 . · arXiv
  19. [Rao and Georgeff, 1995] Anand S Rao and Michael P Georgeff. Bdi agents: From theory to practice. In ICMAS , volume 95, pages 312-319, 1995.
  20. [Ribeiro et al. , 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the Int. Conf.x on Knowledge Discovery and Data Mining , pages 1135-1144. ACM, 2016.
  21. [Slugoski et al. , 1993] Ben R Slugoski, Mansur Lalljee, Roger Lamb, and Gerald P Ginsburg. Attribution in conversational context: Effect of mutual knowledge on explanation-giving. European Journal of Social Psychology , 23(3):219-238, 1993.
  22. [Swartout and Moore, 1993] William R Swartout and Johanna D Moore. Explanation in second generation expert systems. In Second generation expert systems , pages 543-585. Springer, 1993.
  23. [Thagard, 1989] Paul Thagard. Explanatory coherence. Behavioral and Brain Sciences , 12(03):435-467, 1989.

Cited by

Sorted by citations

Authors
Miller, T., Howe, P., & Sonenberg, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1712.00547
arXiv
1712.00547
OpenAlex
W2774522520
Cited by
218
References
6
Citations by year
2026: 9 · 2025: 15 · 2024: 29 · 2023: 28 · 2022: 22 · 2021: 50 · 2020: 29 · 2019: 28 · 2018: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
explainable AI, social and behavioral sciences, user-centered design, philosophy of artificial intelligence, cognitive science, human-AI interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review