The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 17 of the 264 sources its claims cite were checked for support.
Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
Open accessAbstract
Systematic review of automation bias in high-stakes human-AI collaboration, examining cognitive mechanisms, expertise, trust dynamics, verification demands, and the limits of explainable AI for mitigating overreliance.
Overview
CoChat Summary
## TL;DR
This paper is a PRISMA-based review of automation bias in human-AI collaboration, synthesizing 35 peer-reviewed experimental studies from 2015–April 2025. It concludes that trust, task complexity, AI literacy, expertise, and individual differences shape overreliance, while explainability/transparency alone often do not prevent automation bias and can sometimes worsen it.
## Key findings
- The review included 35 studies, published across 21 journals, with a combined 19,774 participants; all included studies used quantitative, experimental methods.
- Trust and task complexity emerged as the main drivers of automation bias; the paper reports that trust accounted for up to 24.1% of the variance in reliant behavior in one study (Küper et al. 2025).
- Explainability was mixed or limited: CFFs reduced reliance on incorrect AI recommendations compared with simple XAI, but were seen as more cognitively demanding; multiple explanation types did not improve detection of incorrect AI recommendations, and trust calibration feedback did not help.
- The review reports several quantitative examples of bias effects, including Bond et al. (2018) where incorrect automated diagnoses reduced ECG interpretation accuracy from 84.86% to 41.66% for fellows and from 86.38% to 27.43% for non-fellows, and Dratsch et al. (2023) where mammography accuracy dropped from 79.7% to 19.8%, 81.3% to 24.8%, and 82.3% to 45.5% across experience groups when AI predictions were incorrect.
## Methods
- Systematic review conducted using PRISMA 2020; the authors defined five research questions on causal/mediator factors, co-occurring biases, explainability, AI literacy/mental models, and individual differences.
- Searches were run in SCOPUS, ScienceDirect, PubMed, and Google Scholar in May 2025, using peer-reviewed English-language studies from January 2015 to April 2025 that were quantitative, experimental, and focused on automation bias or over-reliance.
- Screening proceeded from title/abstract to full text, with 35 studies included; the two reviewers reported 0.95 Cohen’s kappa agreement.
## Limitations
- The paper states that restricting inclusion to peer-reviewed journal articles from 2015–2025, English-language studies, and experimental designs may have excluded earlier work, non-English studies, preprints, qualitative studies, and mixed-methods research.
- The paper states its search strategy lacked sensitivity and missed some relevant studies due to indexing gaps and terminology variation.
- The paper notes limited evidence on cultural and socio-demographic factors because of the current lack of cross-cultural comparison studies.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
17 Valid · 3 Not valid · 203 Not checked · 48 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To the best of our knowledge, existing reviews have focused on AB in relation to traditional computer-based aids rather than AI-driven systems (see Goddard etal. 2012; Lyell and Coiera 2017).”
What the sources say
- Not matchedcites Goddard etal. 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Lyell and Coiera 2017 sourceExperimental — accuracy not yet measured
The Introduction and Method describe a review of automation bias involving clinical decision support and other automated aids, which supports the characterization of this review's focus. Our copy does not distinguish those aids from AI-driven systems or establish that existing reviews generally excluded AI-driven systems. (LLM, conf=0.82)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Cognitive overload is considered the most significant factor inhibiting users from adequately verifying automation accuracy, thereby increasing AB:”
What the sources say
- Partly supportedcites Lyell and Coiera 2017 sourceExperimental — accuracy not yet measured
The Discussion hypothesizes that cognitive overload can prevent users from adequately verifying automation and lead to automation-bias errors. It does not establish cognitive overload as the most significant factor; the Conclusion says further research is needed to test its impact. The supplied passages do not include page numbers. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Users often fail to critically evaluate automation outputs when they perceive the system to be highly reliable (Lyell and Coiera 2017).”
What the sources say
- Supportedcites Lyell and Coiera 2017 source
The Introduction (page unavailable) says high system accuracy may foster trust and that users with greater trust in automation are less likely to detect its failures. It also describes automation bias as replacing vigilant information seeking with reliance on automated cues, supporting the paraphrase. (LLM, conf=0.90)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the harder the verification complexity, that is the complexity of verifying that the automation is performing correctly, the greater the likelihood of undue trust in the system (Lyell and Coiera 2017).”
What the sources say
- Partly supportedcites Lyell and Coiera 2017 sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says high verification complexity appears to increase the risk of automation bias, and the Method defines verification complexity as the difficulty of checking whether automation performs correctly. The claim presents that tentative relationship as a definite one; the paper does not establish that likelihood rises consistently as complexity increases. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In line with these results, Dikmen and Burns (2022) found that providing domain-specific knowledge helped non-expert users to critically evaluate AI outputs, relying less on incorrect AI predictions.”
What the sources say
- Supportedcites Dikmen and Burns 2022 source
Section 6.1 (page unavailable) reports that participants given domain-relevant guidance made fewer investments when the AI underpredicted loan-default risk. It says they may have noticed discrepancies and questioned the AI’s guidance, and concludes that conveying domain knowledge reduced over-reliance on incorrect AI predictions. The finding concerns the peer-to-peer lending task studied. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
132 more claims · 9 with their cited sources checked
Create a free account to see all 271 claims and every source verdict.
- It may state two separate claims as one. 96
- Probably not a substantial claim. 55
- Borderline: one of the tests for a substantial claim is unclear. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A comparable dynamic may plausibly extend to generative AI used as an information source, where fluent and confident outputs can be accepted uncritically (Romeo and Conti 2026), although this transfer across system classes should be treated with caution.”
From Epistemic Trust in Generative AI as an Information Source: Development and Validation of the Trust in AI-Epistemic Scale (TAI-E) (2026) · cites this paper as Romeo and Conti 2026
In the citing paper
“This configuration does not fit either of two simplified narratives that frequently appear in the surrounding literature: the assumption that analytic engagement necessarily reduces trust (Alon-Barkat and Busuioc 2023; Romeo and Conti 2026;Stanovich et al. 2016), or the assumption that trust primarily reflects uncritical acceptance (Davis 1989;Hoff and Bashir 2015).”
From Epistemic Trust in Generative AI as an Information Source: Development and Validation of the Trust in AI-Epistemic Scale (TAI-E) (2026) · cites this paper as Romeo and Conti 2026
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“In the automation context, trust is defined as «the attitude that an (automated) agent will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability» (Lee and See 2004: p.51).”
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714 · cited 2×
“In accordance with PRISMA-S guidelines (Rethlefsen et al. 2021), we also included records identified through personal files.”
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“Key challenges lie in AI literacy, whose lack can limit user’s ability to effectively collaborate with systems (Long and Magerko 2020), and explanation design itself.”
Complacency and Bias in Human Use of Automation: An Attentional Integration
Parasuraman, R., & Manzey, D. H. (2010) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720810376055 · Cited by 1,710 · cited 1×
“Some researchers emphasize differences in their cognitive and situational antecedents (Wickens et al. 2015), while others highlight substantial overlap, viewing both as manifestations of shared underlying attentional and decision-making mechanisms (Parasuraman and Manzey 2010).”
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 7×
“Additionally, accountability, which has been proposed as an intervention to mitigate AB (Goddard et al. 2012), lacks empirical support.”
InfluentialBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124 · cited 12×
“In an effort to reduce over-reliance, Buçinca et al. (2021) adopted CFFs, a metacognitive debiasing technique aimed at promoting analytical thinking.”
InfluentialZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · cited 1×
“Merely making AI explainable or transparent is not enough: explanations must actually help users recognize uncertainty or errors (Zhang et al. 2020).”
Research works printed but not linked: 2
- Hemmer P, Schemmer M, Vössing M, Kühl N (2021) Human–AI complementarity in hybrid intelligence systems: a structured literature review. In: Pacific Asia conference on information systems, vol 78. ISBN 978-1-7336325-7-7
- Romeo G, Conti D (2024) Beyond automation: reshaping Human-artificial intelligence interaction. Sistemi Intelligenti XXVI(3):641–648. https://doi.org/10.1422/115336 · doi:10.1422/115336
Cited by
Toward a science of human–AI teaming for decision making: a complementarity framework
Gonzalez, C., Donahue, K., Goldstein, D. G., Heidari, H., Jalali, M. S., Schelble, B., Singh, A., & Woolley, A. W. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag030 · Cited by 33
“Teams should establish escalation and disagreement protocols (eg confidence thresholds for deferring to humans; concurrence requirements for critical actions) and treat errors as learning opportunities through after-action reviews that incorporate both human and AI system outputs (109).”
Cognitive Risks of AI: Literacy, Trust, and Critical Thinking
Kulal, A. (2025) · Journal of Computer Information Systems · doi:10.1080/08874417.2025.2582050 · Cited by 45
Watts, K. J. (2025) · Education Sciences · doi:10.3390/educsci15101304 · Cited by 18
Cognitive Atrophy Paradox of AI–Human Interaction: From Cognitive Growth and Atrophy to Balance
Kabashkin, I. (2025) · Information · doi:10.3390/info16111009 · Cited by 38
“Human–AI collaboration studies extend these insights, showing that overconfidence in AI may increase even among moderately experienced users, especially when transparency is low, stakes are ambiguous, or monitoring demands are reduced [22].”
Examining human reliance on artificial intelligence in decision making
Pearson, J., Dror, I. E., Jayes, E., Whordley, G.-R., Mason, G., & Nightingale, S. J. (2026) · Scientific Reports · doi:10.1038/s41598-026-34983-y · Cited by 32
“much work has focused on the aviation sector, concerns about automation bias extend to other domains such as healthcare and military, especially with the relatively recent move to embed AI systems within these areas to assist human decision making 12,13 .”
Metadata
- Authors
- Romeo, G., & Conti, D.
- Affiliations
- University of Catania (IT)
- Corresponding author
- Romeo, Giuseppe
- Conti, Daniela
- Venue
- AI & SOCIETY
- Published
- 2025-07-03
- Volume
- 41
- Issue
- 1
- Pages
- 259-278
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W4412013201
- Cited by
- 209
- References
- 60
- Citations by year
- 2026: 194 · 2025: 15
- FWCI
- 228.31
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- AI & Society (publishedVersion, cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- automation bias, human-AI collaboration, explainable AI, trust in AI, AI literacy, explanation complexity, professional expertise, explanation design, user engagement, cognitive mechanisms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring automation bias in human–AI collaboration: a review and implications for explainable AI. https://app.cochat.ai/papers/10.1007/s00146-025-02422-7
- The paper
- Romeo, G., & Conti, D. (2026). Exploring automation bias in human–AI collaboration: a review and implications for explainable AI. AI & SOCIETY. https://doi.org/10.1007/s00146-025-02422-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.