Explanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
The DOI matches its Crossref record; 120 of the 122 references it cites were found in a registry or an index.
Abstract
Information systems with an "intelligent" or "knowledge" component are now prevalent and include knowledge-based systems, decision support systems, intelligent agents, and knowledge management systems. These systems are in principle capable of explaining their reasoning or justifying their behavior. There appears to be a lack of understanding, however, of the benefits that can flow from explanation use, and how an explanation function should be constructed. Work with newer types of intelligent systems and help functions for everyday systems, such as word-processors, appears in many cases to neglect lessons learned in the past. This paper attempts to rectify this situation by drawing together the considerable body of work on the nature and use of explanations. Empirical studies, mainly with knowledge-based systems, are reviewed and linked to a sound theoretical base. The theoretical base combines a cognitive effort perspective, cognitive learning theory, and Toulmin's model of argumentation. Conclusions drawn from the review have both practical and theoretical significance. Explanations are important to users in a number of circumstances— when the user perceives an anomaly, when they want to learn, or when they need a specific piece of knowledge to participate properly in problem solving. Explanations, when suitably designed, have been shown to improve performance and learning and result in more positive user perceptions of a system. The design is important, however, because it appears that explanations will not be used if the user has to exert "too much" effort to get them. Explanations should be provided automatically if this can be done relatively unobtrusively, or by hypertext links, and should be context-specific rather than generic. Explanations that conform to Toulmin's model of argumentation, in that they provide adequate justification for the knowledge offered, should be more persuasive and lead to greater trust, agreement, satisfaction, and acceptance—of the explanation and possibly also of the system as a whole.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While knowledge-based systems can provide clear explanations of the reasoning behind their outputs (Wick & Slagle, 1989; Gregor & Benbasat, 1999), models trained by machine learning algorithms may not (Lipton, 2015).”
From Algorithmic Accountability and Public Reason (2018) · cites this paper as Gregor & Benbasat, 1999
In the citing paper
“Receivers may require explanations about how the AI software came to a decision, especially when they must take action based on that decision (Gregor and Benbasat 1999; Watson et al. 2019).”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Gregor and Benbasat 1999
In the citing paper
“Empirical studies (Gregor and Benbasat 1999) suggest that the suitability of an explanation differs among receivers according to their expertise.”
From How to Design AI for Social Good: Seven Essential Factors (2020) · cites this paper as Gregor and Benbasat 1999
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
User Acceptance of Computer Technology: A Comparison of Two Theoretical Models
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989) · Management Science · doi:10.1287/mnsc.35.8.982 · Cited by 25,709
A BEHAVIORAL MODEL OF RATIONAL CHOICE
Simon, H. A. (1955) · The Quarterly Journal of Economics · doi:10.2307/1884852 · Cited by 15,877
Moore, G. C., & Benbasat, I. (1991) · Information Systems Research · doi:10.1287/isre.2.3.192 · Cited by 9,150
Self-citationHuman agency in social cognitive theory.
Bandura, A. (1989) · American Psychologist · doi:10.1037/0003-066x.44.9.1175 · Cited by 8,263
Toulmin, S. E. (2003) · doi:10.1017/cbo9780511840005 · Cited by 7,101
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology · doi:10.2307/2063712 · Cited by 6,834
Anderson, J. R. (2013) · doi:10.4324/9781315799438 · Cited by 6,761
Newell, A. (1972) · Cited by 6,222
Rational choice and the structure of the environment.
Simon, H. A. (1956) · Psychological Review · doi:10.1037/h0042769 · Cited by 5,919
Research works printed but not linked: 26
- Anderson, 1985, Cognitive Psychology and Its Implications
- Ausubel, 1985, Learning as Constructing Meaning, 71
- Best, 1989, Cognitive Psychology
- Bruner, 1956, A Study of Thinking
- Brunswik, 1952, The Conceptual Foundation of Psychology
- Buchanan, 1984, Expert Systems
- Eining, 1991, The Impact of Expert System Usage on Experiential Learning in an Auditing Setting, Journal of Information Systems, 5, 1
- 1995, Expert Systems with Applications, 8
- Fitts, 1976, Human Performance
- Flowler, 1985, The Relationship Between Cognitive Style and Dialogue Style
- Gregor, S. D. Explanations from Knowledgebased Systems for Human Learning and Problem Solving, unpublished doctoral dissertation, University of Queensland, Brisbane, Australia, 1996a.
- Gregor, 1997, Explanations from Knowledgebased Systems and the Learning versus Working Argument
- Gregor, 1997, Knowledge-based Systems in Prescriptive versus Supportive Roles and the Use of Explanations
- Kidd, 1985, The Consultative Role of an Expert System, 228
- Koonce, 1992, Explanation and Counterexplanation During Audit Analytical Review, The Accounting Review, 67, 59
- Lerch, 1993, Measuring Trust in Machine Advice
- 1981, The Macquarie Dictionary
- Mao, 1996, The Effects of Hypertext-based Explanations in Knowledgebased Systems on Explanation Use and Knowledge Transfer
- Mao, 1996, Exploring the Use of Explanations in Knowledge-based Systems: A Process Tracing Analysis
- Murphy, 1990, Expert System Use and the Development of Expertise in Auditing: A Preliminary Investigation, Journal of Information Systems, 18
- Oltman, 1971, Group Embedded Figures Test
- Schank, 1986, Explanation: A First Pass, 139
- Schroder, 1967, Human Information Processing
- Seligman, 1990, Learned Optimism
- Wognum, P. M. Explanation of Automated Reasoning: How and Why?unpublished doctoral dissertation, University of Twente, Enschede, The Netherlands, 1990.
- Yourdon, 1989, Modern Structure Analysis
Cited by
Items 501–582 of 582
Goram, M., & Veiel, D. (2020) · Advances in human and social aspects of technology book series · doi:10.4018/978-1-7998-4894-3.ch002
Gek, D. K., Кукарцев, В. В., Маркевич, И. В., Shalaeva, D., Yadrov, A. S., & Rogova, D. (2021) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/1151/1/012042
“The user interface provides the transfer of information by the user of the application and the software and hardware complex [6, 8, 11].”
MethodStrategic Information System Agility: From Theory to Practices: References
(2020) · Strategic Information System Agility: From Theory to Practices · doi:10.1108/978-1-80043-810-120211011
Mobile Purchase Decision Support Systems for In-Store Shopping Environments
Kowatsch, T., & Maaß, W. (2012) · IGI Global eBooks · doi:10.4018/978-1-4666-2455-9.ch069
How People Make Decisions and Take Action
(2012) · IGI Global eBooks · doi:10.4018/978-1-4666-0152-9.ch013
Sim, J. (2012) · PsycEXTRA Dataset · doi:10.1037/e525192013-009
(2012) · IGI Global eBooks · doi:10.4018/978-1-4666-0131-4.ch006
An Information Systems Design Theory for Digital Broker Platforms
Bley, K., Hentschel, R., & Pappas, I. O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-06516-3_1
Gregor, S. D. (2023) · Communications of the Association for Information Systems · doi:10.17705/1cais.05207
Self-citationFloridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0009
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0001
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0004
Mapping the Ethics of Algorithms
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0007
How to Deliver a Good AI Society: Some Recommendations
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0010
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0007
Bad Practices: The Misuse of AI for Social Evil
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0008
List of Most Common Abbreviations and Acronyms
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0011
The Gambit: AI Impact on Climate Change
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0011
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0010
Good Practices: The Proper Use of AI for Social Good
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0009
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.002.0005
Conclusion: The Green and the Blue
Floridi, L. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780198883098.003.0013
Threats or Opportunities? Enhancing Firm Performance in the Era of Generative AIPreprint
Lee, S. J., Lehmann, J., Chae, H., Shin, D.-H., Auh, S., & Han, S. P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4771103
Schlepper, Y., Lutz, B., Lindenmeier, J., & Neumann, D. (2024) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-58396-4_3
A qualitative investigation of explanation goodness in a biomedical AI system
Johs, A. J., Weber, R. O., & Agosto, D. E. (2024) · iDEA - Drexel University Institutional Repository · doi:10.17918/00010668 · Cited by 1
Rebholz, T. R., Koop, A., & Hütter, M. (2024) · Technology Mind and Behavior · doi:10.1037/tmb0000136.supp
Three Essays on Artificial Intelligence Adoption and Use
Nguyen, Q. (2019) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc1538732
Füßl, A., & Nissen, V. (2025) · Forum Dienstleistungsmanagement · doi:10.1007/978-3-658-48325-8_13 · Cited by 1
Pivoting Towards user-Centric Implementation of Business Analytics in Information SystemsPreprint
Sobrie, L., Roets, B., Verschelde, M., & Triantis, K. P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5383812
Amitai, Y., Mirsky, R., & Amir, O. (2025) · IEEE International Conference on AI and Data Analytics (ICAD) · doi:10.1109/icad65464.2025.11114061
“Similarly, unhelpful or irrelevant explanations fail to address users’ needs, limiting their utility [23].”
IMAT: Uma Ferramenta para Análise de Modelos de Aprendizado de Máquina Interpretáveis
Assis, A., Dantas, J. R., & Andrade, E. (2025) · Anais do Brazilian Workshop on Social Network Analysis and Mining (BraSNAM) · doi:10.5753/brasnam.2025.8924
Creating Value from Enterprise AI
Someh, I., & Wixom, B. H. (2025) · Springer eBooks · doi:10.1007/978-3-032-01940-0_8
Harmony in autonomy : Navigating the collaboration between humans and autonomous AI agents
Lämmermann, L. (2025) · EPub Bayreuth (University of Bayreuth) · doi:10.15495/epub_ubt_00008369 · Cited by 1
Human-AI Interaction – Investigating the Impact on Individuals and Organizations
Peters, F. (2023) · TUbilio (Technical University of Darmstadt) · doi:10.26083/tuprints-00023070
Visual Representation of Explainable Artificial Intelligence Methods: Design and Empirical Studies
Martínez, M. A. M. (2023) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000164577
Von Requirements zu Privacy Explanations: Ein nutzerzentrierter Ansatz für Usable Privacy
Brunotte, W. (2024) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/16335
Automated provenance collection of runtime model evolution to enable explanation
Reynolds, O. J. (2024) · Aston Publications Explorer (Aston University) · doi:10.48780/publications.aston.ac.uk.00047437
Toniolo, K. (2024) · AMS Dottorato Institutional Doctoral Theses Repository (University of Bologna) · doi:10.48676/unibo/amsdottorato/11427
Berenfeld, N., Nan, N., & WOO, C. C. (2025) · Journal of the Association for Information Systems
Explainable AI for Human-AI Joint Decision-Making in Farming Practice
Hüllmann, J. A. (2022) · University of Twente Research Information
Sîrbu, A.-M., Schelhorn, T. C., & Gnewuch, U. (2025) · AIS Electronic Library (AISeL) · Cited by 2
The New Generation of Recommendation Agents (RAs 2.0): An Affordance Perspective
Wang, J. F. (2023) · VTechWorks (Virginia Tech) · Cited by 1
Can we do better explanations? A proposal of user-centered explainable AI
Ribera, M., & Lapedriza, A. (2019) · The Oberta in Open Access (Universitat Oberta de Catalunya)
Weller, N., Schmitt, A., Fahse, T., & Blohm, I. (2025) · AIS Electronic Library (AISeL) · Cited by 2
Clement, J. (2023) · University of Minnesota Digital Conservancy (University of Minnesota)
Abdelhalim, E. (2023) · MacSphere (McMaster University) · doi:10.71548/13620 · Cited by 2
Li, A. T., De, L., & Ye, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5533921 · Cited by 2
Teaming with AI: Human Behavior in AI-Assisted Decision-Making
Papenkordt, J. (2025) · Amtliche Mitteilungen (Universitätsbibliothek Paderborn) · doi:10.17619/unipb/1-2433 · Cited by 2
Analysis of Large Language Model Prompting and Generation Using Toulmin’s Model
Dalzell, G., Liu, C., Peng, W., & Yu, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4969-6_1
Jacquemin, P., Graf, M., Mehler, M., Walenta, D., Hendriks, P., & Buxmann, P. (2025) · Journal of Intellectual Capital · doi:10.1108/jic-03-2025-0090
Heinisuo, E. (2025) · Tampere University Institutional Repository (Tampere University)
A study of explanation generation in a rule-based system
Mernissi, K. E. (2017) · theses.fr (ABES) · doi:10.70675/1b699650z4c0fz40d3zaf8ez92049a1eb915 · Cited by 1
“The proposition of Johnson and Johnson (1993) is followed by a great deal of empirical researches, for instance as in (Gregor and Benbasat, 1999), describing the factors a ecting explanation usage linked to their theoretical foundations.”
BackgroundLLM-Enhanced Modeling of Social Desirability-Aware Forced-Choice Personality Assessment
Tu, Y., Shi, H., & Zheng, C. (2026) · Electronics · doi:10.3390/electronics15091792
Füßl, A., & Nissen, V. (2026) · Contributions to management science · doi:10.1007/978-3-032-17384-3_13
AI in Financial Reporting and Corporate Liability
Dao, T. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-6390-5.ch012
Explaining Facial Expression Recognition for Human Understanding and Model PersonalisationPreprint
Nahulanthran, S., Vered, M., Kulić, D., & Tian, L. (2026) · Research Square · doi:10.21203/rs.3.rs-9582396/v1
Towards a Design Theory of Discourse Strategies for Conversational Agents
Gregor, S. D., Maedche, A., & Morana, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28316-0_13
Self-citationTalafidaryani, M., & JÚNIOR, S. M. (2026) · Digital Transformation and Society · doi:10.1108/dts-05-2025-0108
Sałabun, W., Kizielewicz, B., Bączkiewicz, A., & Wątróbski, J. (2026) · Journal of the Association for Information Systems
CLARO: Constrained budget allocation with rule-based decision interpretation
Rezvanjoo, H. (2026) · SoftwareX · doi:10.1016/j.softx.2026.102935
Kojan, L., Riepen, C., Mortaga, M., Ponge, J., Suer, J., & Valdez, A. S. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29839-3_11
Kumar, R., & Dhankhar, P. (2026) · Multimedia Research · doi:10.46253/j.mr.v9i2.a4
Rahman, W., & Sharmin, S. (2026) · Strategic Business Research · doi:10.1016/j.sbr.2026.100261 · Cited by 1
Parker, M., Pietersen, H., Sewpaul, T., Maart, R., & Joubert, J. (2026) · Research Square · doi:10.21203/rs.3.rs-10090979/v1
人工知能時代における第二言語学習:認知-AIリテラシー-読解統合モデル(CAR)
Sun, Z. (2026) · Institutional Repositories DataBase (IRDB) · doi:10.14989/doctor.k26811
Explainability as a Social Cue: Social Perception, Trust, and Adoption Intent in Conversational AIPreprint
Sauka, K., Venniker, M., Kackovic, M., & Situmeang, F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7395560
Mao, C., Shi, H., Jia, H., & DiFranzo, D. J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.09713
“If users interpret the rationale as indicating that the assistant can connect its outputs to reasons, they may evaluate the assistant as more knowledgeable, thoughtful, or intelligent [13, 50].”
AI Explanations Through the IT Artifact Views
Vatn, D. M. K., & Mikalef, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38014-2_28
Alloatti, F. (2022) · Institutional Research Information System University of Turin (University of Turin)
Olukayode, O., Natarajan, P., Widiastuti, C. T., & Wibowo, S. (2026) · Data Journal of Information Systems and Management · doi:10.61978/data.v4i2.2002
“Gregor and Benbasat (1999) distinguished between systems that provide no explanation, those that provide simple trace explanations, and those that provide strategic explanations of why a recommendation was generated; they predicted that higher-quality explanations would increase user trust and…”
A Multi-faceted Evaluation of LLM Explanations for AI Credit Scoring Models
Sachs, T. A., & Rodríguez-Serrano, J. A. (n.d.) · Journal of the Association for Information Systems
Algorithmic Antidotes: Heuristic-Aligned Explanations for Debiasing Judgment
Richardson, B., Karahanna, E., & Schecter, A. (n.d.) · Journal of the Association for Information Systems
Zhang, Y., Gong, Y., Wang, K., Sun, R., So, S. L., & Li, J. (2026) · Journal of Global Information Management · doi:10.4018/jgim.425948
From Explaining to Enabling: Actionable Transparency in Algorithmic Insurance PricingPreprint
Machado, A. F., & Huang, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.34459
Explanation and Validation as Conditions for Trust in Judicial AI in Bangladesh
Ferdous, M., Maisha, I. J., Maisha, I. J., & Mannan, K. A. (2026) · Journal of State Government and Mass Media · doi:10.64907/xkmf.v04i04.jsgmm.1
Tracing truth: How traceability enhances fake news detection design
Brachten, F., Kühl, N., Eisenhardt, D., & Meske, C. (2026) · Electronic Markets · doi:10.1007/s12525-026-00938-4
From Human to Hybrid: Reinventing Dual Concern Theory in the Age of AI Transformation
Bidmeshk, O. G., & Kohli, A. (2026) · Telematics and Informatics Reports · doi:10.1016/j.teler.2026.100391
Jain, A. (2006) · Journal of the Association for Information Systems
HEALTH ANALYTICS AND PREDICTIVE MODELING: FOUR ESSAYS ON HEALTH INFORMATICS
Lin, Y.-K. (2015) · UA Campus Repository (The University of Arizona)
Rapp, H. P. (2012) · Bradford Scholars (University of Bradford) · Cited by 5
Metadata
- Authors
- Gregor, S., & Benbasat, I.
- Affiliations
- Central Queensland University (AU)
- University of British Columbia (CA)
- Venue
- MIS Quarterly
- Publisher
- MIS Quarterly
- Published
- 1999-12-01
- Volume
- 23
- Issue
- 4
- Pages
- 497-530
- ISSN
- 0276-7783 (ISSN-L)
- 2162-9730
- OpenAlex
- W1576620340
- Cited by
- 535
- References
- 122
- Citations by year
- 2026: 28 · 2025: 31 · 2024: 20 · 2023: 47 · 2022: 22 · 2021: 43 · 2020: 32 · 2019: 30 · 2018: 21 · 2017: 21 · 2016: 13 · 2015: 11 · 2014: 17 · 2013: 6 · 2012: 22
- FWCI
- 7.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- MIS Quarterly (publishedVersion)
- Journal of the Association for Information Systems (submittedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- explanation generation, intelligent systems, expert systems, decision support system, intelligent agents, knowledge management systems, Toulmin model, cognitive effort, cognitive learning theory, explanation design, user trust, user satisfaction, system acceptance, word processing software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.