Explanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
The DOI matches its Crossref record; 120 of the 122 references it cites were found in a registry or an index.
Abstract
Information systems with an "intelligent" or "knowledge" component are now prevalent and include knowledge-based systems, decision support systems, intelligent agents, and knowledge management systems. These systems are in principle capable of explaining their reasoning or justifying their behavior. There appears to be a lack of understanding, however, of the benefits that can flow from explanation use, and how an explanation function should be constructed. Work with newer types of intelligent systems and help functions for everyday systems, such as word-processors, appears in many cases to neglect lessons learned in the past. This paper attempts to rectify this situation by drawing together the considerable body of work on the nature and use of explanations. Empirical studies, mainly with knowledge-based systems, are reviewed and linked to a sound theoretical base. The theoretical base combines a cognitive effort perspective, cognitive learning theory, and Toulmin's model of argumentation. Conclusions drawn from the review have both practical and theoretical significance. Explanations are important to users in a number of circumstances— when the user perceives an anomaly, when they want to learn, or when they need a specific piece of knowledge to participate properly in problem solving. Explanations, when suitably designed, have been shown to improve performance and learning and result in more positive user perceptions of a system. The design is important, however, because it appears that explanations will not be used if the user has to exert "too much" effort to get them. Explanations should be provided automatically if this can be done relatively unobtrusively, or by hypertext links, and should be context-specific rather than generic. Explanations that conform to Toulmin's model of argumentation, in that they provide adequate justification for the knowledge offered, should be more persuasive and lead to greater trust, agreement, satisfaction, and acceptance—of the explanation and possibly also of the system as a whole.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While knowledge-based systems can provide clear explanations of the reasoning behind their outputs (Wick & Slagle, 1989; Gregor & Benbasat, 1999), models trained by machine learning algorithms may not (Lipton, 2015).”
From Algorithmic Accountability and Public Reason (2018) · cites this paper as Gregor & Benbasat, 1999
In the citing paper
“Receivers may require explanations about how the AI software came to a decision, especially when they must take action based on that decision (Gregor and Benbasat 1999; Watson et al. 2019).”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Gregor and Benbasat 1999
In the citing paper
“Empirical studies (Gregor and Benbasat 1999) suggest that the suitability of an explanation differs among receivers according to their expertise.”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Gregor and Benbasat 1999
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
User Acceptance of Computer Technology: A Comparison of Two Theoretical Models
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989) · Management Science · doi:10.1287/mnsc.35.8.982 · Cited by 25,709
A BEHAVIORAL MODEL OF RATIONAL CHOICE
Simon, H. A. (1955) · The Quarterly Journal of Economics · doi:10.2307/1884852 · Cited by 15,877
Moore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150
Self-citationHuman agency in social cognitive theory.
Bandura, A. (1989) · American Psychologist · doi:10.1037/0003-066x.44.9.1175 · Cited by 8,263
Toulmin, S. E. (2003) · doi:10.1017/cbo9780511840005 · Cited by 7,101
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology · doi:10.2307/2063712 · Cited by 6,834
Anderson, J. R. (2013) · doi:10.4324/9781315799438 · Cited by 6,761
Newell, A. (1972) · Cited by 6,222
Rational choice and the structure of the environment.
Simon, H. A. (1956) · Psychological Review · doi:10.1037/h0042769 · Cited by 5,919
Research works printed but not linked: 26
- Anderson, 1985, Cognitive Psychology and Its Implications
- Ausubel, 1985, Learning as Constructing Meaning, 71
- Best, 1989, Cognitive Psychology
- Bruner, 1956, A Study of Thinking
- Brunswik, 1952, The Conceptual Foundation of Psychology
- Buchanan, 1984, Expert Systems
- Eining, 1991, The Impact of Expert System Usage on Experiential Learning in an Auditing Setting, Journal of Information Systems, 5, 1
- 1995, Expert Systems with Applications, 8
- Fitts, 1976, Human Performance
- Flowler, 1985, The Relationship Between Cognitive Style and Dialogue Style
- Gregor, S. D. Explanations from Knowledgebased Systems for Human Learning and Problem Solving, unpublished doctoral dissertation, University of Queensland, Brisbane, Australia, 1996a.
- Gregor, 1997, Explanations from Knowledgebased Systems and the Learning versus Working Argument
- Gregor, 1997, Knowledge-based Systems in Prescriptive versus Supportive Roles and the Use of Explanations
- Kidd, 1985, The Consultative Role of an Expert System, 228
- Koonce, 1992, Explanation and Counterexplanation During Audit Analytical Review, The Accounting Review, 67, 59
- Lerch, 1993, Measuring Trust in Machine Advice
- 1981, The Macquarie Dictionary
- Mao, 1996, The Effects of Hypertext-based Explanations in Knowledgebased Systems on Explanation Use and Knowledge Transfer
- Mao, 1996, Exploring the Use of Explanations in Knowledge-based Systems: A Process Tracing Analysis
- Murphy, 1990, Expert System Use and the Development of Expertise in Auditing: A Preliminary Investigation, Journal of Information Systems, 18
- Oltman, 1971, Group Embedded Figures Test
- Schank, 1986, Explanation: A First Pass, 139
- Schroder, 1967, Human Information Processing
- Seligman, 1990, Learned Optimism
- Wognum, P. M. Explanation of Automated Reasoning: How and Why?unpublished doctoral dissertation, University of Twente, Enschede, The Netherlands, 1990.
- Yourdon, 1989, Modern Structure Analysis
Cited by
The Nature of Theory in Information Systems1
Gregor, S. (2006) · MIS Quarterly · doi:10.2307/25148742 · Cited by 3,306
“…(see, for example, Achinstein 1983; Craik 1943; Hempel and Oppenheim 1948; Nagel 1979), in relation to everyday reasoning and the nature of argumentation (see Toulmin 1958; Toulmin et al. 1979), as well as in the subbranches of IS relating to knowledge-based systems (see Gregor and Benbasat 1999).”
BackgroundSelf-citationSynthesizing information systems knowledge: A typology of literature reviews
Paré, G., Trudel, M.-C., Jaana, M., & Kitsiou, S. (2015) · Information & Management · doi:10.1016/j.im.2014.08.008 · Cited by 1,987
E-Commerce Product Recommendation Agents: Use, Characteristics, and Impact1
Xiao, B., & Benbasat, I. (2007) · MIS Quarterly · doi:10.2307/25148784 · Cited by 1,081
“…to involve actual use of an RA (prototype or operational, web-based or stand-alone) by human users (similar to the definition of empirical studies in Gregor and Benbasat 1999) in either online or physical settings.8 • The study must have dependent variables that go beyond “accuracy” and…”
BackgroundInfluentialSelf-citationQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
“n kind of explanation...to satisfy certain user purposes of user goals. ” In the early generation of AI work, question-driven frameworks were used to generate explanations for knowledge-based systems [20,38,88]. AQUA [72] is a reasoning model that uses questions to generate explanations and identify knowledge gaps for…”
MethodManaging artificial intelligence
Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16274 · Cited by 898
Metadata
- Authors
- Gregor, S., & Benbasat, I.
- Affiliations
- Central Queensland University (AU)
- University of British Columbia (CA)
- Venue
- MIS Quarterly
- Publisher
- MIS Quarterly
- Published
- 1999-12-01
- Volume
- 23
- Issue
- 4
- Pages
- 497-530
- ISSN
- 0276-7783 (ISSN-L)
- 2162-9730
- OpenAlex
- W1576620340
- Cited by
- 535
- References
- 122
- Citations by year
- 2026: 28 · 2025: 31 · 2024: 20 · 2023: 47 · 2022: 22 · 2021: 43 · 2020: 32 · 2019: 30 · 2018: 21 · 2017: 21 · 2016: 13 · 2015: 11 · 2014: 17 · 2013: 6 · 2012: 22
- FWCI
- 7.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- MIS Quarterly (publishedVersion)
- Journal of the Association for Information Systems (submittedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- explanation generation, intelligent systems, expert systems, decision support system, intelligent agents, knowledge management systems, Toulmin model, cognitive effort, cognitive learning theory, explanation design, user trust, user satisfaction, system acceptance, word processing software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.