CoChat
Sign Up
Cited by 256Artificial Intelligence in Healthcare and EducationCardiovascular Function and Risk FactorsArtificial Intelligence in Healthcare

Blinded, randomized trial of sonographer versus AI cardiac function assessment

Open access
He, Bryan, Kwan, Alan C., Cho, Jae Hyung, Yuan, Neal, Pollick, Charles, Shiota, Takahiro, Ebinger, Joseph, Bello, Natalie A. +8 more·2023·Nature
DOI 10.1038/s41586-023-05947-3PMC PMC10115627
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence (AI) has been developed for echocardiography 1–3 , although it has not yet been tested with blinding and randomization. Here we designed a blinded, randomized non-inferiority clinical trial (ClinicalTrials.gov ID: NCT05140642; no outside funding) of AI versus sonographer initial assessment of left ventricular ejection fraction (LVEF) to evaluate the impact of AI in the interpretation workflow. The primary end point was the change in the LVEF between initial AI or sonographer assessment and final cardiologist assessment, evaluated by the proportion of studies with substantial change (more than 5% change). From 3,769 echocardiographic studies screened, 274 studies were excluded owing to poor image quality. The proportion of studies substantially changed was 16.8% in the AI group and 27.2% in the sonographer group (difference of −10.4%, 95% confidence interval: −13.2% to −7.7%, P < 0.001 for non-inferiority, P < 0.001 for superiority). The mean absolute difference between final cardiologist assessment and independent previous cardiologist assessment was 6.29% in the AI group and 7.23% in the sonographer group (difference of −0.96%, 95% confidence interval: −1.34% to −0.54%, P < 0.001 for superiority). The AI-guided workflow saved time for both sonographers and cardiologists, and cardiologists were not able to distinguish between the initial assessments by AI versus the sonographer (blinding index of 0.088). For patients undergoing echocardiographic quantification of cardiac function, initial assessment of LVEF by AI was non-inferior to assessment by sonographers.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41586-023-05947-3

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 34 · Cited by 256

Research works printed but not linked: 28

  1. [2]Duffy G High-throughput precision phenotyping of left ventricular hypertrophy with cardiovascular deep learning JAMA 2022 7 386 395 PMC9008505 35195663
  2. [4]Heidenreich PA 2022 AHA/ACC/HFSA Guideline for the Management of Heart Failure: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines Circulation 2022 145 e895 e1032 35363499
  3. [5]Dunlay SM Roger VL Redfield MM Epidemiology of heart failure with preserved ejection fraction Nat. Rev. Cardiol. 2017 14 591 602 10.1038/nrcardio.2017.65 28492288 · DOI
  4. [6]Al-Khatib SM 2017 AHA/ACC/HRS Guideline for Management of Patients with Ventricular Arrhythmias and the Prevention of Sudden Cardiac Death: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart Rhythm Society Circulation 2018 138 e272 e391 29084731
  5. [7]Wilcox JE Fang JC Margulies KB Mann DL Heart failure with recovered left ventricular ejection fraction: JACC Scientific Expert Panel J. Am. Coll. Cardiol. 2020 76 719 734 10.1016/j.jacc.2020.05.075 32762907 · DOI
  6. [8]Yuan N Systematic quantification of sources of variation in ejection fraction calculation using deep learning JACC Cardiovasc. Imaging 2021 14 2260 2262 10.1016/j.jcmg.2021.06.018 34274282 PMC10697700 · DOI
  7. [9]Pellikka PA Variability in ejection fraction measured by echocardiography, gated single-photon emission computed tomography, and cardiac magnetic resonance in patients with coronary artery disease and left ventricular dysfunction JAMA Netw. Open 2018 1 e181456 10.1001/jamanetworkopen.2018.1456 30646130 PMC6324278 · DOI · Paper page
  8. [10]Lang RM Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging Eur. Heart J. Cardiovasc. Imaging 2015 16 233 270 10.1093/ehjci/jev014 25712077 · DOI
  9. [11]Cole GD Defining the real-world reproducibility of visual grading of left ventricular function and visual estimation of left ventricular ejection fraction: impact of image quality, experience and accreditation Int. J. Cardiovasc. Imaging 2015 31 1303 1314 10.1007/s10554-015-0659-1 26141526 PMC4572050 · DOI
  10. [12]Shahgaldi K Gudmundsson P Manouras A Brodin L-A Winter R Visually estimated ejection fraction by two dimensional and triplane echocardiography is closely correlated with quantitative ejection fraction by real-time three dimensional echocardiography Cardiovasc. Ultrasound 2009 7 41 10.1186/1476-7120-7-41 19706183 PMC2747837 · DOI
  11. [13]Stone JR Kanneganti R Abbasi M Akhtari M Monitoring for chemotherapy-related cardiotoxicity in the form of left ventricular systolic dysfunction: a review of current recommendations JCO Oncol. Pract. 2021 17 228 236 10.1200/OP.20.00924 33689453 · DOI
  12. [14]Chen JH Asch SM Machine learning and prediction in medicine—beyond the peak of inflated expectations N. Engl. J. Med. 2017 376 2507 2509 10.1056/NEJMp1702071 28657867 PMC5953825 · DOI · Paper page
  13. [15]Huang M-S Wang C-S Chiang J-H Liu P-Y Tsai W-C Automated recognition of regional wall motion abnormalities through deep neural network interpretation of transthoracic echocardiography Circulation 2020 142 1510 1520 10.1161/CIRCULATIONAHA.120.047530 32964749 · DOI
  14. [16]Wu E How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals Nat. Med. 2021 27 582 584 10.1038/s41591-021-01312-x 33820998 · DOI · Paper page
  15. [17]Schiano C Machine learning and bioinformatics framework integration to potential familial DCM-related markers discovery Genes 2021 12 1946 10.3390/genes12121946 34946895 PMC8701745 · DOI
  16. [18]Infante T Radiogenomics and artificial intelligence approaches applied to cardiac computed tomography angiography and cardiac magnetic resonance for precision medicine in coronary heart disease: a systematic review Circ. Cardiovasc. Imaging 2021 14 1133 1146 10.1161/CIRCIMAGING.121.013025 34915726 · DOI
  17. [19]Bazoukis G The inclusion of augmented intelligence in medicine: a framework for successful implementation Cell Rep. Med. 2022 3 100485 10.1016/j.xcrm.2021.100485 35106506 PMC8784713 · DOI
  18. [20]Bang H Ni L Davis CE Assessment of blinding in clinical trials Control. Clin. Trials 2004 25 143 156 10.1016/j.cct.2003.10.016 15020033 · DOI · Paper page
  19. [21]Yao X Artificial intelligence-enabled electrocardiograms for identification of patients with low ejection fraction: a pragmatic, randomized clinical trial Nat. Med. 2021 27 815 819 10.1038/s41591-021-01335-4 33958795 · DOI · Paper page
  20. [22]Attia ZI An artificial intelligence-enabled ECG algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction Lancet 2019 394 861 867 10.1016/S0140-6736(19)31721-0 31378392 · DOI · Paper page
  21. [23]Noseworthy PA Artificial intelligence-guided screening for atrial fibrillation using electrocardiogram during sinus rhythm: a prospective non-randomised interventional trial Lancet 2022 400 1206 1212 10.1016/S0140-6736(22)01637-3 36179758 · DOI · Paper page
  22. [24]Farsalinos KE Head-to-head comparison of global longitudinal strain measurements among nine different vendors: the EACVI/ASE inter-vendor comparison study J. Am. Soc. Echocardiogr. 2015 28 1171 1181.e2 10.1016/j.echo.2015.06.011 26209911 · DOI
  23. [25]Persell SD Effect of home blood pressure monitoring via a smartphone hypertension coaching application or tracking application on adults with uncontrolled hypertension: a randomized clinical trial JAMA Netw. Open 2020 3 e200255 10.1001/jamanetworkopen.2020.0255 32119093 PMC7052730 · DOI · Paper page
  24. [26]Vodrahalli, K., Daneshjou, R., Gerstenberg, T. & Zou, J. Do humans trust advice more if it comes from AI? In Proc. 2022 AAAI/ACM Conference on AI, Ethics, and Society 763–777 (Association for Computing Machinery, 2022).
  25. [28]Cheitlin MD ACC/AHA Guidelines for the Clinical Application of Echocardiography. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Clinical Application of Echocardiography). Developed in collaboration with the American Society of Echocardiography Circulation 1997 95 1686 1744 10.1161/01.CIR.95.6.1686 9118558 · DOI
  26. [29]Douglas PS Echocardiographic imaging in clinical trials: American Society of Echocardiography Standards for echocardiography core laboratories: endorsed by the American College of Cardiology Foundation J. Am. Soc. Echocardiogr. 2009 22 755 765 10.1016/j.echo.2009.05.020 19560654 · DOI
  27. [30]Douglas PS 2019 ACC/AHA/ASE Key Data Elements and definitions for Transthoracic Echocardiography: a report of the American College of Cardiology/American Heart Association Task Force on clinical data standards (Writing Committee to Develop Clinical Data Standards for Transthoracic Echocardiography) and the American Society of Echocardiography Circ. Cardiovasc. Imaging 2019 12 e000027 10.1161/HCI.0000000000000027 31233331 · DOI
  28. [31]CONSORT-AI and SPIRIT-AI Steering Group. Reporting guidelines for clinical trials evaluating artificial intelligence interventions are needed Nat. Med. 2019 25 1467 1468 10.1038/s41591-019-0603-3 31551578 · DOI · Paper page

Authors
He, B., Kwan, A. C., Cho, J. H., Yuan, N., Pollick, C., Shiota, T., Ebinger, J., Bello, N. A., Wei, J., Josan, K., Duffy, G., Jujjavarapu, M., Siegel, R., Cheng, S., Zou, J. Y., & Ouyang, D.
Affiliations
  • Stanford University (US)
  • Cedars-Sinai Medical Center (US)
  • University of California, San Francisco (US)
Corresponding author
  • Cheng, Susan
  • Zou, James Y.
  • Ouyang, David
ORCID
Venue
Nature
Publisher
Springer Science and Business Media LLC
Published
2023-04-05
Volume
616
Issue
7957
Pages
520-524
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
DOI
10.1038/s41586-023-05947-3
PMC
PMC10115627
OpenAlex
W4362608384
Cited by
256
References
34
Citations by year
2026: 73 · 2025: 77 · 2024: 72 · 2023: 34
FWCI
10.22
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • Cedars-Sinai Medical Center
  • National Heart, Lung, and Blood Institute (R01HL173526, R00HL157421)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Cardiovascular Function and Risk Factors
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
artificial intelligence, echocardiography, left ventricular ejection fraction, cardiac function assessment, non-inferiority, time savings
MeSH terms
Cardiologists, Artificial Intelligence, Echocardiography, Heart Function Tests, Humans, Stroke Volume, Reproducibility of Results, Single-Blind Method, Ventricular Function, Left, Workflow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review