The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing
Open accessAbstract
Although future regulations increasingly advocate that AI applications must be interpretable by users, we know little about how such explainability can affect human information processing. By conducting two experimental studies, we help to fill this gap. We show that explanations pave the way for AI systems to reshape users' understanding of the world around them. Specifically, state-of-the-art explainability methods evoke mental model adjustments that are subject to confirmation bias, allowing misconceptions and mental errors to persist and even accumulate. Moreover, mental model adjustments create spillover effects that alter users' behavior in related but distinct domains where they do not have access to an AI system. These spillover effects of mental model adjustments risk manipulating user behavior, promoting discriminatory biases, and biasing decision making. The reported findings serve as a warning that the indiscriminate use of modern explainability methods as an isolated measure to address AI systems' black-box problems can lead to unintended, unforeseen problems because it creates a new channel through which AI systems can influence human behavior in various domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Trust, Reciprocity, and Social History
Berg, J., Dickhaut, J., & McCabe, K. (1995) · Games and Economic Behavior · doi:10.1006/game.1995.1027 · Cited by 5,548
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 1×
“Feature-based XAI methods’ potential to link different domains is particularly concerning given recent evidence on their susceptibility to intentional manipulation and adversarial attacks (Lipton 2018).”
BackgroundDegroot, M. H. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10480137 · Cited by 4,004
The Nature of Theory in Information Systems1
Gregor, S. (2006) · MIS Quarterly · doi:10.2307/25148742 · Cited by 3,306 · cited 1×
“Although providing explanations, in general, can have a variety of cognitive effects, researchers across disciplines generally agree that they primarily enhance people’s understanding of someone or something, improve reasoning, and facilitate learning (Gregor 2006, Malle 2006).”
BackgroundAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 1×
“…of humans and AI systems including the perceived subjectivity of the task (Castelo et al. 2019, Logg et al. 2019), seeing the system err (Dietvorst et al. 2015), being able to modify predictions (Dietvorst et al. 2018), the divergence bet-ween actual and expected predictive performance…”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 3×
“We opted to use these explanations because they are already widely in use in practice and because there are arguments that feature-based explanations on an individual level are necessary to comply with (upcoming) regulatory requirements (Goodman and Flaxman 2017).”
BackgroundMethodAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
XAI—Explainable artificial intelligence
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G.-Z. (2019) · Science Robotics · doi:10.1126/scirobotics.aay7120 · Cited by 2,053
Schön, D. A. (2017) · doi:10.4324/9781315237473 · Cited by 1,740
Task-Dependent Algorithm Aversion
Castelo, N., Bos, M. W., & Lehmann, D. R. (2019) · Journal of Marketing Research · doi:10.1177/0022243719851788 · Cited by 1,500 · cited 1×
“Several factors moderate the interaction of humans and AI systems including the perceived subjectivity of the task (Castelo et al. 2019, Logg et al. 2019), seeing the system err (Dietvorst et al. 2015), being able to modify predictions (Dietvorst et al. 2018), the divergence bet-ween actual and…”
BackgroundTask-Dependent Algorithm Aversion
Castelo, N., Bos, M. W., Bos, M. W., Lehmann, D. R., & Lehmann, D. R. (2019) · Springer eBooks · doi:10.1007/978-3-031-88067-4_21 · Cited by 1,441
Festinger, L. (1962) · Scientific American · doi:10.1038/scientificamerican1062-93 · Cited by 1,421
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364 · cited 2×
“This effect could reinforce the already considerable influence contemporary AI systems have on human societies (Rahwan et al. 2019) by, for better or worse, allowing human users to adopt systems’ inner logic and problem-solving strategies.”
BackgroundDietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 1×
“…subjectivity of the task (Castelo et al. 2019, Logg et al. 2019), seeing the system err (Dietvorst et al. 2015), being able to modify predictions (Dietvorst et al. 2018), the divergence bet-ween actual and expected predictive performance (Jus-supow et al. 2020), and, most importantly for our…”
BackgroundOn looking into the black box: Prospects and limits in the search for mental models.
Rouse, W. B., & Morris, N. M. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.3.349 · Cited by 1,218
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024
First Impressions Matter: A Model of Confirmatory Bias
Rabin, M., & Schrag, J. L. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399555945 · Cited by 1,001
Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019) · Business & Information Systems Engineering · doi:10.1007/s12599-019-00595-2 · Cited by 826 · cited 1×
“…(Goldstein et al. 2017, Case 2018) documents how hybrid human-AI systems (i.e., cen-taur systems) achieve superior results in comparison with the entities operating independently (Dellermann et al. 2019, Tschandl et al. 2020), promising considerable benefits from successful human-AI collaboration.”
ResultVarieties of Confirmation Bias
Klayman, J. (1995) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60315-1 · Cited by 755
Noise : a flaw in human judgment
Kahneman, D. J., Sibony, O., & Sunstein, C. R. (2021) · Cited by 749
Pyszczynski, T., & Greenberg, J. (1987) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60417-7 · Cited by 635
Editorial—Big Data, Data Science, and Analytics: The Opportunity and Challenge for IS Research
Agarwal, R., & Dhar, V. (2014) · Information Systems Research · doi:10.1287/isre.2014.0546 · Cited by 610
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · cited 3×
“Given the inherent distinctions between expert systems and ML-based AI systems in terms of encoded knowledge, contemporary explainability methods present an entirely different form of reasoning to users, namely that of machines (Vilone and Longo 2021, Meske et al. 2022).”
BackgroundExplanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
Gregor, S., & Benbasat, I. (1999) · MIS Quarterly · doi:10.2307/249487 · Cited by 535
Knobloch-Westerwick, S., & Meng, J. (2009) · Communication Research · doi:10.1177/0093650209333030 · Cited by 500
Recommendation Agents for Electronic Commerce: Effects of Explanation Facilities on Trusting Beliefs
Wang, W., & Benbasat, I. (2007) · Journal of Management Information Systems · doi:10.2753/mis0742-1222230410 · Cited by 498
Beyond Deterrence: An Expanded View of Employee Computer Abuse1
Willison, R., & Warkentin, M. (2013) · MIS Quarterly · doi:10.25300/misq/2013/37.1.01 · Cited by 479
Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16553 · Cited by 469
Why we still use our heads instead of formulas: Toward an integrative approach.
Kleinmuntz, B. (1990) · Psychological Bulletin · doi:10.1037//0033-2909.107.3.296 · Cited by 437
How the mind explains behavior: folk explanations, meaning, and social interaction
(2005) · Choice Reviews Online · doi:10.5860/choice.42-6170 · Cited by 436
Jussupow, E., Spohrer, K., Heinzl, A., & Gawlitza, J. (2021) · Information Systems Research · doi:10.1287/isre.2020.0980 · Cited by 422
Estimating the Impact of “Humanizing” Customer Service Chatbots
Schanke, S., Burtch, G., & Ray, G. (2021) · Information Systems Research · doi:10.1287/isre.2021.1015 · Cited by 398
Yin, D., Mitra, S., & Zhang, H. (2016) · Information Systems Research · doi:10.1287/isre.2015.0617 · Cited by 365
Explainable artificial intelligence: Objectives, stakeholders, and future research opportunities
Meske, C., Bunde, E., Schneider, J., & Gersch, M. (2022) · Information Systems Management · doi:10.1080/10580530.2020.1849465 · Cited by 365
When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring
van den Broek, E., Sergeeva, A., & Huysman, M. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16559 · Cited by 330
Exploring the impact of artificial Intelligence: Prediction versus judgment
Agrawal, A., Gans, J. S., & Goldfarb, A. (2019) · Information Economics and Policy · doi:10.1016/j.infoecopol.2019.05.001 · Cited by 311
SHAP and LIME: An Evaluation of Discriminative Power in Credit Risk
Gramegna, A., & Giudici, P. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.752558 · Cited by 306 · cited 6×
“Notably, LIME and SHAP closely relate to the seminal description of Gregor and Ben-basat (1999) of “why and why not explanations” in the context of knowledge-based expert systems.”
BackgroundInfluentialMachine Learning and Portfolio Optimization
Ban, G.-Y., El Karoui, N., & Lim, A. E. B. (2018) · Management Science · doi:10.1287/mnsc.2016.2644 · Cited by 282
A Comparative Study of Distributed Learning Environments on Learning Outcomes
Alavi, M., Marakas, G. M., & Yoo, Y. (2002) · Information Systems Research · doi:10.1287/isre.13.4.404.72 · Cited by 274
Hoffman, M., Kahn, L. B., & Li, D. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx042 · Cited by 261
Explainability in human–agent systems
Rosenfeld, A., & Richardson, A. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09408-y · Cited by 224
Vandenbosch, B., & Higgins, C. (1996) · Information Systems Research · doi:10.1287/isre.7.2.198 · Cited by 202
Dhaliwal, J. S., & Benbasat, I. (1996) · Information Systems Research · doi:10.1287/isre.7.3.342 · Cited by 194
Failures of Fairness in Automation Require a Deeper Understanding of Human–ML Augmentation
Teodorescu, M. H. M., Morse, L., Awwad, Y., & Kane, G. C. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16535 · Cited by 194
Chatterjee, S., Sarker, S., & Valacich, J. S. (2015) · Journal of Management Information Systems · doi:10.1080/07421222.2014.1001257 · Cited by 187
Cognitive dissonance: Reexamining a pivotal theory in psychology (2nd ed.).
(2019) · American Psychological Association eBooks · doi:10.1037/0000135-000 · Cited by 186
Rico-Juan, J. R., & Paz, P. T. D. L. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114590 · Cited by 173
An update on Bayesian updating
Holt, C. A., & Smith, A. M. (2009) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2007.08.013 · Cited by 158
Mao, I. B. J.-Y. (2000) · Journal of Management Information Systems · doi:10.1080/07421222.2000.11045646 · Cited by 146
Human–Robot Interaction: When Investors Adjust the Usage of Robo-Advisors in Peer-to-Peer Lending
Ge, R., Zheng, Z., Tian, X., & Liao, L. (2021) · Information Systems Research · doi:10.1287/isre.2021.1009 · Cited by 130
Lim, K. H., Ward, L. M., & Benbasat, I. (1997) · Information Systems Research · doi:10.1287/isre.8.3.254 · Cited by 125
User agreement with incorrect expert system advice
Dijkstra, J. J. (1999) · Behaviour and Information Technology · doi:10.1080/014492999118832 · Cited by 124
Expl(AI)n It to Me – Explainable AI and Information Systems Research
Bauer, K., Hinz, O. F., van der Aalst, W. M. P., & Weinhardt, C. (2021) · Business & Information Systems Engineering · doi:10.1007/s12599-021-00683-2 · Cited by 94
Self-citationCognitive dissonance and utility maximization
Gilad, B., Kaish, S., & Loeb, P. D. (1987) · Journal of Economic Behavior & Organization · doi:10.1016/0167-2681(87)90021-7 · Cited by 71
Revealed Preferences in a Sequential Prisoners' Dilemma: A Horse-Race between Six Utility Functions
Miettinen, T., Kosfeld, M., Fehr, E., & Weibull, J. W. (2020) · Journal of Economic Behavior & Organization · doi:10.2139/ssrn.2939010 · Cited by 43
Human-Machine Collaboration in Cancer and Beyond: The Centaur Care Model
Goldstein, I. M., Lawrence, J., & Miner, A. S. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2016.6413 · Cited by 27
Cited by
Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise
Chen, Z., & Chan, J. (2024) · Management Science · doi:10.1287/mnsc.2023.03014 · Cited by 204
Williamson, S. M., & Prybutok, V. (2024) · Information · doi:10.3390/info15060299 · Cited by 146
Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
“…highlight prompt changes and the resulting output changes (Zamfirescu-Pereira et al., 2023), use explain-ability features indicating factors that influenced the output (Bauer et al., 2023), or use an analytic dashboard that shows statistics of AI tools’ performance changes (Wang et al., 2024).”
Kruschel, S., Hambauer, N., Weinzierl, S., Zilker, S., Kraus, M., & Zschech, P. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00922-2 · Cited by 50
Kabir, S., Hossain, M. S., & Andersson, K. (2025) · Algorithms · doi:10.3390/a18090556 · Cited by 44
Metadata
- Authors
- Bauer, K., von Zahn, M., & Hinz, O.
- Venue
- Information Systems Research
- Published
- 2023-12
- Volume
- 34
- Issue
- 4
- Pages
- 1582-1602
- ISSN
- 1047-7047 (ISSN-L)
- 1526-5536
- OpenAlex
- W4323048183
- Cited by
- 261
- References
- 58
- FWCI
- 22.58
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.