Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; 9 of the 31 sources its claims cite were checked for support.

Do as AI say: susceptibility in deployment of clinical decision-aids

Open access

Abstract

Artificial intelligence (AI) models for decision support have been developed for clinical settings such as radiology, but little work evaluates the potential impact of such systems. In this study, physicians received chest X-rays and diagnostic advice, some of which was inaccurate, and were asked to evaluate advice quality and make diagnoses. All advice was generated by human experts, but some was labeled as coming from an AI system. As a group, radiologists rated advice as lower quality when it appeared to come from an AI system; physicians with less task-expertise did not. Diagnostic accuracy was significantly worse when participants received inaccurate advice, regardless of the purported source. This work raises important considerations for how advice, AI and non-AI, should be deployed in clinical environments.

Overview

CoChat Summary

TL;DR

In a pre-registered web experiment, physicians reviewing chest X-rays received diagnostic advice that was always generated by human experts but sometimes labeled as coming from an AI system. Inaccurate advice reduced diagnostic accuracy regardless of purported source, while radiologists rated purported AI advice as lower quality than purported human advice.

Key findings

  • Radiologists showed algorithmic aversion in advice-quality ratings: they rated advice significantly lower when it was labeled AI versus human (γ = 0.53, p = 0.001), whereas IM/EM physicians did not (γ = 0.22, p = 0.165).
  • Inaccurate advice was rated lower in quality by both groups, more strongly by radiologists (γ = −1.00, p < 0.001) than by IM/EM physicians (γ = −0.40, p = 0.002).
  • Diagnostic accuracy was significantly worse with inaccurate advice for radiologists (β = 2.39, p < 0.001) and IM/EM physicians (β = 1.91, p < 0.001); task experts performed 40.10% better and non-experts 37.53% better with accurate versus inaccurate advice.
  • Purported source did not significantly affect diagnostic accuracy (radiologists p = 0.183; IM/EM p = 0.081). Susceptibility varied: 41.73% of IM/EM physicians always gave the wrong diagnosis with inaccurate advice versus 27.54% of radiologists; 28.26% of radiologists and 17.32% of IM/EM physicians refuted all incorrect advice.

Methods

  • Pre-registered web-based experiment (Qualtrics) with physicians recruited in the US and Canada: 138 radiologists (high task expertise) and 127 internal/emergency medicine physicians (lower task expertise).
  • Participants reviewed eight chest X-ray cases from the MIMIC Chest X-ray database, each with a clinical vignette and diagnostic advice; advice was human-generated but labeled as either from an AI system (CHEST-AI) or an experienced radiologist (Dr. S. Johnson).
  • The design was a 2 × 2 mixed factorial: advice source (AI vs. human) between subjects and advice accuracy (accurate vs. inaccurate) within subjects, with participants randomly assigned six accurate and two inaccurate cases. Separate linear mixed models for advice-quality ratings and logistic mixed models for diagnostic accuracy regressed outcomes on advice accuracy, source, their interaction, and individual-related covariates, with a random effect by participant.

Limitations

  • The paper states that only eight cases were tested to keep the study short enough for physician completion, that the web-based tool may not capture real deployment risk or behavior, that physician conversations are difficult to recreate, and that the sample is skewed toward less experienced physicians.
  • The paper notes that advice was always generated by human experts and only the purported source was manipulated, so the study does not test actual AI-generated advice.
  • Evident from the text: quality ratings may not reflect actual attitudes or behavior, as radiologists’ expressed aversion to AI advice did not affect their reliance on it; the online format may also have led to less careful case review or greater reliance on advice.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-021-00385-9

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of the experiment. Each participant reviewed eight cases. For each case, the physician would see the chest X-ray as well as diagnostic advice, which would either be accurate or inaccurate. The advice was labeled as coming either from an AI system or an experienced radiologist. Participants were then asked to rate the quality of the advice and make a final diagnosis.
Fig. 1 Overview of the experiment. Each participant reviewed eight cases. For each case, the physician would see the chest X-ray as well as diagnostic advice, which would either be accurate or inaccurate. The advice was labeled as coming either from an AI system or an experienced radiologist. Participants were then asked to rate the quality of the advice and make a final diagnosis.CC BY · PubMed Central
Fig. 2 Advice quality rating across advice accuracy and source. We demonstrate the effect of the accuracy of advice and source of advice on the quality rating across both types of physicians: task experts (radiologists), and non-experts (IM/EM physicians). In (a) we compare clinical advice ratings across accuracy, demonstrating that while both groups rated accurate advice as high-quality, only task experts rated inaccurate advice as low-quality. In (b) we compare clinical advice ratings across source, demonstrating that only task the experts rated purported human advice as significantly higher quality. There is no significant interaction between advice accuracy and advice source. The boxplots show 25th to 75th percentiles (lower and upper hinges) with the median depicted by the central line; the whiskers extend to a maximum of 1.5× interquartile range (IQR) beyond the boxes. *p ≤ 0.05, **p ≤ 0.001, ns = not significant.
Fig. 2 Advice quality rating across advice accuracy and source. We demonstrate the effect of the accuracy of advice and source of advice on the quality rating across both types of physicians: task experts (radiologists), and non-experts (IM/EM physicians). In (a) we compare clinical advice ratings across accuracy, demonstrating that while both groups rated accurate advice as high-quality, only task experts rated inaccurate advice as low-quality. In (b) we compare clinical advice ratings across source, demonstrating that only task the experts rated purported human advice as significantly higher quality. There is no significant interaction between advice accuracy and advice source. The boxplots show 25th to 75th percentiles (lower and upper hinges) with the median depicted by the central line; the whiskers extend to a maximum of 1.5× interquartile range (IQR) beyond the boxes. *p ≤ 0.05, **p ≤ 0.001, ns = not significant.CC BY · PubMed Central
Fig. 3 Diagnostic accuracy across advice accuracy and source. We demonstrate the effect of the accuracy of advice and source of advice on diagnostic accuracy for task experts (radiologists) and non-experts (IM/EM physicians). In (a) we compare diagnostic accuracy across advice accuracy, demonstrating that both groups perform better when they receive accurate advice. In (b) we compare diagnostic accuracy across advice sources, demonstrating that neither group of physicians had a significant difference in diagnostic accuracy depending on the source of advice. There is no significant interaction between advice accuracy and advice source. The error bars represent confidence intervals. *p ≤ 0.05, **p ≤ 0.001, ns = not significant.
Fig. 3 Diagnostic accuracy across advice accuracy and source. We demonstrate the effect of the accuracy of advice and source of advice on diagnostic accuracy for task experts (radiologists) and non-experts (IM/EM physicians). In (a) we compare diagnostic accuracy across advice accuracy, demonstrating that both groups perform better when they receive accurate advice. In (b) we compare diagnostic accuracy across advice sources, demonstrating that neither group of physicians had a significant difference in diagnostic accuracy depending on the source of advice. There is no significant interaction between advice accuracy and advice source. The error bars represent confidence intervals. *p ≤ 0.05, **p ≤ 0.001, ns = not significant.CC BY · PubMed Central
Fig. 4 Individual performance. We show the individual performance of radiologists (a) and IM/EM physicians (b) sorted in increasing order by the number of cases they correctly diagnosed. Each physician’s individual performance is split into cases with accurate advice (the lower, blue part of the bar) and inaccurate advice (the upper, red part of the bar). We further indicate Critical Performers, who always recognize inaccurate advice, and Susceptible Performers, who never do.
Fig. 4 Individual performance. We show the individual performance of radiologists (a) and IM/EM physicians (b) sorted in increasing order by the number of cases they correctly diagnosed. Each physician’s individual performance is split into cases with accurate advice (the lower, blue part of the bar) and inaccurate advice (the upper, red part of the bar). We further indicate Critical Performers, who always recognize inaccurate advice, and Susceptible Performers, who never do.CC BY · PubMed Central
Fig. 5 Case performance. Individual case performance amongst radiologists and IM/EM physicians. Each participant reviewed all eight cases; case order and advice accuracy was randomized per participant.
Fig. 5 Case performance. Individual case performance amongst radiologists and IM/EM physicians. Each participant reviewed all eight cases; case order and advice accuracy was randomized per participant.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 1 Not valid · 10 Not checked · 48 Own results and interpretations

22 more claims

Create a free account to see all 63 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 19

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

13 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentr(1102) = 0.43reported p < .001 · recomputed p < .001

    “r(1102) = 0.43, p < .001”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 42 references

Cited by

See all 543

Metadata

Authors
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M.
Affiliations
  • University of Regensburg (DE)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Massachusetts Institute of Technology (US)
  • Boston Medical Center (US)
  • Beth Israel Deaconess Medical Center (US)
  • FOM University of Applied Sciences for Economics and Management (DE)
  • St. Michael's Hospital (CA)
  • University of Toronto (CA)
Corresponding author
  • Gaube, Susanne
  • Suresh, Harini
Venue
npj Digital Medicine
Published
2021-02-19
Volume
4
Issue
1
Pages
31-31
ISSN
2398-6352 (ISSN-L)
OpenAlex
W3130937151
Cited by
543
References
42
Citations by year
2026: 174 · 2025: 109 · 2024: 81 · 2023: 94 · 2022: 62 · 2021: 23
FWCI
18.71
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • University of Regensburg Publication Server (University of Regensburg) (submittedVersion)
Funders
  • Gouvernement du Canada | Natural Sciences and Engineering Research Council of Canada
  • Microsoft Research
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
artificial intelligence, clinical decision support, radiologists, chest X-ray diagnosis, diagnostic accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Do as AI say: susceptibility in deployment of clinical decision-aids. https://app.cochat.ai/papers/10.1038/s41746-021-00385-9
The paper
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021). Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digital Medicine. https://doi.org/10.1038/s41746-021-00385-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.