Cited by 127Clinical Reasoning and Diagnostic SkillsECG Monitoring and AnalysisArtificial Intelligence in Healthcare and Education

Automation bias in medicine: The influence of automated diagnoses on interpreter accuracy and uncertainty when reading electrocardiograms

Open access
Bond, Raymond R., Novotny, Tomas, Andrsova, Irena, Koc, Lumir, Sisakova, Martina, Finlay, Dewar, Guldenring, Daniel, McLaughlin, James, Peace, Aaron, McGilligan, Victoria, Leslie, Stephen J., Wang, Hui, Malik, Marek·2018·Journal of Electrocardiology

Abstract

Introduction Interpretation of the 12‑lead Electrocardiogram (ECG) is normally assisted with an automated diagnosis (AD), which can facilitate an 'automation bias' where interpreters can be anchored. In this paper, we studied, 1) the effect of an incorrect AD on interpretation accuracy and interpreter confidence (a proxy for uncertainty), and 2) whether confidence and other interpreter features can predict interpretation accuracy using machine learning. Methods This study analysed 9000 ECG interpretations from cardiology and non-cardiology fellows (CFs and non-CFs). One third of the ECGs involved no ADs, one third with ADs (half as incorrect) and one third had multiple ADs. Interpretations were scored and interpreter confidence was recorded for each interpretation and subsequently standardised using sigma scaling. Spearman coefficients were used for correlation analysis and C5.0 decision trees were used for predicting interpretation accuracy using basic interpreter features such as confidence, age, experience and designation. Results Interpretation accuracies achieved by CFs and non-CFs dropped by 43.20% and 58.95% respectively when an incorrect AD was presented (p < 0.001). Overall correlation between scaled confidence and interpretation accuracy was higher amongst CFs. However, correlation between confidence and interpretation accuracy decreased for both groups when an incorrect AD was presented. We found that an incorrect AD disturbs the reliability of interpreter confidence in predicting accuracy. An incorrect AD has a greater effect on the confidence of non-CFs (although this is not statistically significant it is close to the threshold, p = 0.065). The best C5.0 decision tree achieved an accuracy rate of 64.67% (p < 0.001), however this is only 6.56% greater than the no-information-rate. Conclusion Incorrect ADs reduce the interpreter's diagnostic accuracy indicating an automation bias. Non-CFs tend to agree more with the ADs in comparison to CFs, hence less expert physicians are more effected by automation bias. Incorrect ADs reduce the interpreter's confidence and also reduces the predictive power of confidence for predicting accuracy (even more so for non-CFs). Whilst a statistically significant model was developed, it is difficult to predict interpretation accuracy using machine learning on basic features such as interpreter confidence, age, reader experience and designation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jelectrocard.2018.08.007

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 21 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Kuhn, 2008, Caret package, J Stat Softw, 28, 1

Cited by

Sorted by citations

Authors
Bond, R. R., Novotny, T., Andrsova, I., Koc, L., Sisakova, M., Finlay, D., Guldenring, D., McLaughlin, J., Peace, A., McGilligan, V., Leslie, S. J., Wang, H., & Malik, M.
Affiliations
  • University of Ulster (GB)
  • Altnagelvin Area Hospital (GB)
  • Raigmore Hospital (GB)
  • Imperial College London (GB)
Corresponding author
Bond, Raymond R.
Venue
Journal of Electrocardiology
Publisher
Elsevier BV
Published
2018-11
Volume
51
Issue
6
Pages
S6-S11
ISSN
0022-0736 (ISSN-L)
OpenAlex
W2887520110
Cited by
127
References
21
Citations by year
2026: 30 · 2025: 19 · 2024: 14 · 2023: 14 · 2022: 25 · 2021: 9 · 2020: 10 · 2019: 6
FWCI
7.13
Type
journal article
Language
en
Open access
Locations
  • Journal of Electrocardiology (publishedVersion)
  • PubMed (publishedVersion)
  • St George's Online Research Archive (St George's University of London) (acceptedVersion)
  • Ulster University Research Portal (Ulster University) (submittedVersion, other-oa)
  • Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
Funders
  • European Union's INTERREG VA
  • Ministry of Health, Czech Republic (65269705)
Topics
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
automation bias, automated diagnosis, ECG interpretation, 12-lead ECG, diagnostic accuracy, uncertainty, anchoring effect, cardiology fellows
MeSH terms
Arrhythmias, Cardiac, Automation, Clinical Competence, Decision Trees, Diagnostic Errors, Electrocardiography, Humans, Observer Variation, Bias, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review