The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; 9 of the 31 sources its claims cite were checked for support.
Do as AI say: susceptibility in deployment of clinical decision-aids
Open accessAbstract
Artificial intelligence (AI) models for decision support have been developed for clinical settings such as radiology, but little work evaluates the potential impact of such systems. In this study, physicians received chest X-rays and diagnostic advice, some of which was inaccurate, and were asked to evaluate advice quality and make diagnoses. All advice was generated by human experts, but some was labeled as coming from an AI system. As a group, radiologists rated advice as lower quality when it appeared to come from an AI system; physicians with less task-expertise did not. Diagnostic accuracy was significantly worse when participants received inaccurate advice, regardless of the purported source. This work raises important considerations for how advice, AI and non-AI, should be deployed in clinical environments.
Overview
CoChat Summary
TL;DR
In a pre-registered web experiment, physicians reviewing chest X-rays received diagnostic advice that was always generated by human experts but sometimes labeled as coming from an AI system. Inaccurate advice reduced diagnostic accuracy regardless of purported source, while radiologists rated purported AI advice as lower quality than purported human advice.
Key findings
- Radiologists showed algorithmic aversion in advice-quality ratings: they rated advice significantly lower when it was labeled AI versus human (γ = 0.53, p = 0.001), whereas IM/EM physicians did not (γ = 0.22, p = 0.165).
- Inaccurate advice was rated lower in quality by both groups, more strongly by radiologists (γ = −1.00, p < 0.001) than by IM/EM physicians (γ = −0.40, p = 0.002).
- Diagnostic accuracy was significantly worse with inaccurate advice for radiologists (β = 2.39, p < 0.001) and IM/EM physicians (β = 1.91, p < 0.001); task experts performed 40.10% better and non-experts 37.53% better with accurate versus inaccurate advice.
- Purported source did not significantly affect diagnostic accuracy (radiologists p = 0.183; IM/EM p = 0.081). Susceptibility varied: 41.73% of IM/EM physicians always gave the wrong diagnosis with inaccurate advice versus 27.54% of radiologists; 28.26% of radiologists and 17.32% of IM/EM physicians refuted all incorrect advice.
Methods
- Pre-registered web-based experiment (Qualtrics) with physicians recruited in the US and Canada: 138 radiologists (high task expertise) and 127 internal/emergency medicine physicians (lower task expertise).
- Participants reviewed eight chest X-ray cases from the MIMIC Chest X-ray database, each with a clinical vignette and diagnostic advice; advice was human-generated but labeled as either from an AI system (CHEST-AI) or an experienced radiologist (Dr. S. Johnson).
- The design was a 2 × 2 mixed factorial: advice source (AI vs. human) between subjects and advice accuracy (accurate vs. inaccurate) within subjects, with participants randomly assigned six accurate and two inaccurate cases. Separate linear mixed models for advice-quality ratings and logistic mixed models for diagnostic accuracy regressed outcomes on advice accuracy, source, their interaction, and individual-related covariates, with a random effect by participant.
Limitations
- The paper states that only eight cases were tested to keep the study short enough for physician completion, that the web-based tool may not capture real deployment risk or behavior, that physician conversations are difficult to recreate, and that the sample is skewed toward less experienced physicians.
- The paper notes that advice was always generated by human experts and only the purported source was manipulated, so the study does not test actual AI-generated advice.
- Evident from the text: quality ratings may not reflect actual attitudes or behavior, as radiologists’ expressed aversion to AI advice did not affect their reliance on it; the online format may also have led to less careful case review or greater reliance on advice.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
4 Valid · 1 Not valid · 10 Not checked · 48 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, it is not clear how to effectively integrate AI tools with human decision-makers; indeed, the few cases where systems have been implemented and studied showed no improved clinical outcomes6,7.”
§ Do as AI say: susceptibility in deployment of clinical decision-aids > Introduction
What the sources say
- Supportedcites [6] Brocklehurst, P. et al. Computerised interpretation of fetal heart rate during labour (INFANT): a randomised controlled trial. Lancet 389, 1719–1729 (2017). source
The Results report no significant difference in poor neonatal outcomes between the decision-support and control groups, and no evidence of differences in secondary outcomes for babies. The Discussion likewise concludes that the software did not reduce poor neonatal outcomes. This trial supports the cited sentence as an example, not as evidence about every AI system. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [7] Nunes, I. et al. Central fetal monitoring with and without computer analysis: a randomized controlled trial. Obstet. Gynecol. 129, 83–90 (2017). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Fig. 5, all cases were impacted by incorrect advice to varying degrees.”
§ Do as AI say: susceptibility in deployment of clinical decision-aids > Results > Incorrect advice impacts some clinical cases more than others
plainstates a causen = 8- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“None of the measured covariates significantly predicted susceptibility, for either radiologists or IM/EM physicians.”
§ Do as AI say: susceptibility in deployment of clinical decision-aids > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Physicians across expertise levels often failed to dismiss inaccurate advice regardless of its source.”
§ Do as AI say: susceptibility in deployment of clinical decision-aids > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, inaccurate advice was more convincing for certain cases that involved additional interaction or analyses to recognize the error.”
§ Do as AI say: susceptibility in deployment of clinical decision-aids > Discussion
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
22 more claims
Create a free account to see all 63 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“it has been shown that people with high task expertise are more inclined to dismiss or devalue task-related advice from an AI system than are people with low task expertise7,18.”
From Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays (2023) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The presentation format of an explanation, its integration into electronic health record (EHR) systems, the cognitive load it imposes, and its alignment with existing clinical reasoning patterns all shape whether a technically sound explanation translates into genuine clinical utility [31,32].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A growing body of work examines how healthcare professionals interact with and respond to AI-generated explanations, a dimension distinct from the technical methods used to produce them.”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Acceptance of clinical AI is shaped by integration into existing workflows, performance under time pressure, alignment with clinical reasoning patterns, and the cognitive load explanations impose [20,31,32].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“studies suggest that convincing explanations may paradoxically increase over-reliance on incorrect AI outputs, particularly among less experienced clinicians [31,32].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [32]
Experimental — accuracy not yet measured
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentr(1102) = 0.43reported p < .001 · recomputed p < .001
“r(1102) = 0.43, p < .001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
The Association of American Medical Colleges
Zapffe, F. C. (1938) · Academic Medicine · doi:10.1097/00001888-193801000-00007 · Cited by 5,533
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Resistance to Medical Artificial Intelligence
Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019) · Journal of Consumer Research · doi:10.1093/jcr/ucz013 · Cited by 2,015
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775
Cited by
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
Abulibdeh, A., Zaidan, E., & Abulibdeh, R. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.140527 · Cited by 641
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548
“When system designers do not take these perceptual differences into consideration it can lead to misinterpretation of output, which is especially problematic in high-risk settings [247, 248].”
BackgroundArtificial Intelligence (AI) Literacy in Early Childhood Education: The Challenges and Opportunities
Su, J., Ng, D. T. K., & Chu, S. K. W. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100124 · Cited by 489
Health system-scale language models are all-purpose prediction engines
Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06160-y · Cited by 460
Metadata
- Authors
- Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M.
- Affiliations
- University of Regensburg (DE)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Massachusetts Institute of Technology (US)
- Boston Medical Center (US)
- Beth Israel Deaconess Medical Center (US)
- FOM University of Applied Sciences for Economics and Management (DE)
- St. Michael's Hospital (CA)
- University of Toronto (CA)
- Corresponding author
- Gaube, Susanne
- Suresh, Harini
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2021-02-19
- Volume
- 4
- Issue
- 1
- Pages
- 31-31
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC7896064
- OpenAlex
- W3130937151
- Cited by
- 543
- References
- 42
- Citations by year
- 2026: 174 · 2025: 109 · 2024: 81 · 2023: 94 · 2022: 62 · 2021: 23
- FWCI
- 18.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- University of Regensburg Publication Server (University of Regensburg) (submittedVersion)
- Funders
- Gouvernement du Canada | Natural Sciences and Engineering Research Council of Canada
- Microsoft Research
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- artificial intelligence, clinical decision support, radiologists, chest X-ray diagnosis, diagnostic accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do as AI say: susceptibility in deployment of clinical decision-aids. https://app.cochat.ai/papers/10.1038/s41746-021-00385-9
- The paper
- Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021). Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digital Medicine. https://doi.org/10.1038/s41746-021-00385-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.