Detecting structural heart disease from electrocardiograms using AI

May combine two claims.

Early detection of structural heart disease improves outcomes, but widespread screening is limited by the cost and accessibility of imaging tools such as echocardiography.

“Early detection of structural heart disease is critical to improving outcomes, but widespread screening remains limited by the cost and accessibility of imaging tools such as echocardiography1,2.”

• cites [1] Otto, C. M. et al. 2020 ACC/AHA guideline for the management of patients with valvular heart disease: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 143, e72–e227 (2021). sourceNot checked — no full text of the source
• cites [2] Heidenreich, P. A. et al. 2022 AHA/ACC/HFSA guideline for the management of heart failure: executive summary: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. J. Am. Coll. Cardiol. 79, 1757–1780 (2022). sourcePartly supported

  Take-Home Message No. 9 (Methods; page unavailable) supports the prevention aspect: it says screening followed by team-based care can help prevent left ventricular dysfunction or new-onset heart failure. It does not say that the cost or accessibility of echocardiography limits widespread screening; searches for those terms found no supporting passage in our copy. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

A study of 2,500 people aged 65 years or older found 4.9% previously diagnosed with clinically significant VHD and another 6.4% with undiagnosed VHD, more than doubling overall prevalence.

“A study of 2,500 people 65 years of age or older found that 4.9% were previously diagnosed with clinically significant (graded moderate or severe) VHD and found another 6.4% with undiagnosed VHD more than doubling the overall prevalence14.”

• cites [14] d’Arcy, J. L. et al. Large-scale community echocardiographic screening reveals a major burden of undiagnosed valvular heart disease in older people: the OxVALVE Population Cohort Study. Eur. Heart J. 37, 3515–3522 (2016). sourcePartly supported

  The Abstract reports 2,500 participants aged ≥65, 6.4% with undiagnosed moderate or severe VHD, and 4.9% with pre-existing VHD, yielding 11.3% overall—more than twice 4.9%. However, the Discussion says the previously diagnosed group was excluded from screening and its clinical significance was assumed, not confirmed by moderate-or-severe grading. Page numbers are not available in the supplied copy. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

May combine two claims.

Finding patients with SHD earlier in the disease process reduces mortality, decreases costs and improves quality of life.

“Finding patients with SHD earlier in the disease process has been shown to reduce mortality, decrease costs and improve quality of life, but getting to diagnosis remains challenging1,15,16.”

• cites [1] Otto, C. M. et al. 2020 ACC/AHA guideline for the management of patients with valvular heart disease: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 143, e72–e227 (2021). sourceNot checked — no full text of the source
• cites [15] Kang, D. H. et al. Early surgery or conservative care for asymptomatic aortic stenosis. N. Engl. J. Med. 382, 111–119 (2020). sourceNot checked — no full text of the source
• cites [16] Suri, R. M. et al. Association between early surgical intervention vs watchful waiting and outcomes for mitral regurgitation due to flail mitral valve leaflets. JAMA 310, 609–616 (2013). sourcePartly supported

  The Conclusion (page unavailable) reports that early surgery after detection of mitral regurgitation due to flail mitral leaflets was associated with greater long-term survival. That supports the mortality component only for this condition and intervention; the paper does not show that finding patients with structural heart disease earlier reduces mortality, nor does it report decreased costs or improved quality of life. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This method of human-machine comparison is similar to typical methods and one we have used previously.

“This method of human–machine comparison is similar to typical methods and one we have used previously34,35.”

• cites [34] Oakden-Rayner, L. & Palmer, L. J. in Artificial Intelligence in Medical Imaging: Opportunities, Applications and Risks (eds Ranschaert, E. R. et al.) 83–104 (Springer, 2019). sourceNot checked — no full text of the source
• cites [35] Bhave, S. et al. Deep learning to detect left ventricular structural abnormalities in chest X-rays. Eur. Heart J. 45, 2002–2012 (2024). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Other ECG-based SHD models, such as rECHOmmend, have demonstrated excellent performance with an AUROC of 91% in its retrospective dataset.

“Other ECG-based SHD models, such as rECHOmmend, have demonstrated excellent performance with an AUROC of 91% in its retrospective dataset4.”

• cites [4] Ulloa-Cerna, A. E. et al. rECHOmmend: an ECG-based machine learning approach for identifying patients at increased risk of undiagnosed structural heart disease detectable by echocardiography. Circulation 146, 36–47 (2022). sourceSupported

May combine two claims.

When EchoNext was trained on data from four of the eight NYP hospitals and tested on independent data from the four unseen hospitals, the performance change was minimal; the same was true when switching the hospitals used for training and testing.

“When the model was trained on data from four of the eight NYP hospitals and tested on independent data from the four unseen hospitals, the performance change was minimal. The same was true when switching the hospitals used for training and testing.”

May combine two claims.

In the silent deployment cohort, 18% (15,094 patients) underwent their first echocardiogram after the ECG, and 38% (5,744 patients) of those were newly diagnosed with SHD.

“In this cohort, 18% (15,094 patients) underwent their first echocardiogram as part of routine clinical care after the ECG; of which, 38% (5,744 patients) were newly diagnosed with SHD.”

May combine two claims.

Right ventricular systolic dysfunction (AUROC 91%) and low left ventricular systolic dysfunction (90%) were the best-performing individual disease components of EchoNext.

“Model performance varied across each of these individual components with right ventricular (AUROC 91%) and low left ventricular systolic dysfunction (90%) as the best performing (Fig. 2 and Supplementary Table 2).”

May combine two claims.

After trial completion, retrospective EchoNext risk stratification of the 100 DISCOVERY patients showed high risk (n = 33) with 24% left-sided VHD and 73% SHD, moderate risk (n = 50) with 2% left-sided VHD and 28% SHD, and low risk (n = 17) with 0% left-sided VHD and 6% SHD.

“The rates of disease were strongly correlated within these groups as follows: high risk (n = 33, 24% with left-sided VHD and 73% with SHD), moderate risk (n = 50, 2% with left-sided VHD, 28% with SHD) and low risk (n = 17, 0% with left-sided VHD and 6% with SHD).”

May combine two claims.

All differences between EchoNext risk groups in DISCOVERY were significant (P = 0.002 for left-sided VHD and P < 0.001 for SHD).

“All differences between risk groups for EchoNext were significant (P = 0.002 for left-sided VHD and P < 0.001 for SHD).”

May combine two claims.

EchoNext had superior accuracy, sensitivity and specificity in detecting SHD from ECGs compared to cardiologists, both when given the AI prediction to consult and when not.

“EchoNext had superior accuracy, sensitivity and specificity in detecting SHD from ECGs compared to cardiologists both when given the AI prediction to consult and when not”

• Also stated in: § Expanding ECG use through AI · boosted

  “the performance of the EchoNext model was significantly superior, improving on both diagnostic sensitivity and specificity compared with the human experts”

May combine two claims.

For the 1,600 non-AI-assisted reviews, cardiologists had accuracy 64.0% (95% CI 61.6-66.4%), sensitivity 61.1% (95% CI 57.3-64.8%) and specificity 66.1% (95% CI 63.0-69.1%).

“For the 1,600 non-AI-assisted reviews, the accuracy of the cardiologists was 64.0% (95% CI 61.6–66.4%) with sensitivity 61.1% (95% CI 57.3–64.8%) and specificity 66.1% (95% CI 63.0–69.1%).”

May combine two claims.

In the 150-ECG survey set, EchoNext had accuracy 77.3%, sensitivity 72.6% and specificity 80.7%.

“In the set of 150 ECGs, the EchoNext model had an accuracy of 77.3%, sensitivity 72.6% and specificity of 80.7%.”

May combine two claims.

Across NYP hospitals, EchoNext AUROC ranged from 82% to 87%, showing stable, generalizable performance.

“Across hospitals (AUROC range 82–87%) (Fig. 3) and clinical contexts (AUROC range 79–84%), the model showed stable, generalizable performance.”

May combine two claims.

There was a significant difference in the number of patients positive for left-sided VHD (P = 0.005) and SHD (P = 0.003) when comparing high- to moderate-risk ValveNet scores.

“There was a significant difference in the number of patients positive for left-sided VHD (P = 0.005) and SHD (P = 0.003) when comparing high- to moderate-risk ValveNet scores.”

May combine two claims.

The lowest EchoNext performance was for low left ventricular wall thickness (AUROC 77%), aortic regurgitation (78%), pulmonary regurgitation (79%) and pericardial effusion (80%).

“The lowest performance was seen for low left ventricular wall thickness (AUROC 77%), AR (78%), pulmonary regurgitation (79%) and pericardial effusion (80%).”

May combine two claims.

EchoNext was prospectively evaluated in a clinical trial of patients without previous cardiac imaging and successfully identified previously undiagnosed heart disease.

“The models were prospectively evaluated in a clinical trial of patients without previous cardiac imaging, successfully identifying previously undiagnosed heart disease.”

May combine two claims.

In DISCOVERY, 47 patients with moderate-risk ValveNet scores had 0% positive for left-sided VHD and 19% positive for SHD.

“A total of 47 patients were recruited with moderate-risk ValveNet scores, with 0% positive for left-sided VHD and 19% positive for SHD.”

May combine two claims.

Cardiologist accuracy differed between clinically normal and abnormal ECGs (69% versus 62%), whereas EchoNext performed equally well (77% for both).

“Notably, the accuracy of the cardiologists differed between clinically normal and abnormal ECGs (69% versus 62%, respectively), whereas EchoNext performed equally well (77% for both) despite the substantial difference in SHD prevalence (25% versus 46.5% in normal versus abnormal ECGs) (Supplementary Table 12).”

May combine two claims.

EchoNext achieved an AUROC of 85.2% (95% CI 84.5-85.9%), AUPRC of 78.5% (95% CI 77.2-79.6%) and diagnostic odds ratio of 12.8 (95% CI 11.6-14.1) on the NYP multicentre test set.

“EchoNext performance on the NYP multicentre test set was high and well calibrated (Figs. 2 and 3 and Supplementary Fig. 1), with an area under the receiver operating characteristic curve (AUROC) of 85.2% (95% confidence interval (CI) 84.5–85.9%), area under the precision–recall curve (AUPRC) of 78.5% (95% CI 77.2–79.6%) and diagnostic odds ratio of 12.8 (95% CI 11.6–14.1) (Fig. 2).”

May combine two claims.

With AI assistance, cardiologist accuracy significantly improved to 69.2% (95% CI 66.9-71.4%), sensitivity 64.7% (95% CI 60.9-68.3%) and specificity 72.4% (95% CI 69.4-75.3%).

“With AI assistance, the accuracy of the cardiologists significantly improved to 69.2% (95% CI 66.9–71.4%) with sensitivity 64.7% (95% CI 60.9–68.3%) and specificity 72.4% (95% CI 69.4–75.3%).”

May combine two claims.

Despite its smaller single-centre training set, the Columbia mini-model's performance was only modestly lower than the full multicentre-trained EchoNext model, with AUROC in the same multicentre test set of 83.1%.

“Of note, despite the smaller and single-centre training set, this model’s performance was only modestly lower than the full multicentre-trained EchoNext model, with AUROC in the same multicentre test set of 83.1%.”

May combine two claims.

At a fixed sensitivity of 70%, external cohorts showed comparable positive predictive value but a 10% drop in specificity.

“At a fixed sensitivity of 70%, the external cohorts showed comparable positive predictive value, but a 10% drop in specificity (Supplementary Table 10).”

May combine two claims.

Using the predefined model score cutoff of 0.6, 27% (4,135 patients) of the ECG and/or echocardiogram population were predicted as high risk, corresponding to precision 74% and recall 53%.

“Using the predefined model score cutoff of 0.6, 27% (4,135 patients) of the ECG and/or echocardiogram population were predicted as high risk, corresponding to a precision (positive predictive value) and recall (sensitivity) of 74% and 53%, respectively.”

May combine two claims.

Across clinical contexts, EchoNext AUROC ranged from 79% to 84%, showing stable performance.

“Across hospitals (AUROC range 82–87%) (Fig. 3) and clinical contexts (AUROC range 79–84%), the model showed stable, generalizable performance.”

May combine two claims.

In DISCOVERY, 53 patients with high-risk ValveNet scores had 17% positive for moderate or greater left-sided VHD and 53% positive for SHD.

“A total of 53 patients with high-risk ValveNet scores were recruited, with 17% positive for moderate or greater left-sided VHD and 53% positive for SHD.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Compared to other models developed for similar pathologies, the current study had significantly increased racial and/or ethnic diversity across many internal or external institutes.

“Compared to other models that have been developed for similar pathologies, the current study had significantly increased racial and/or ethnic diversity across many internal or external institutes.”

May combine two claims.

The SHD label in the current study captures 98.9% of all echo-based diagnoses as compared to 65.6% by rECHOmmend.

“the SHD label in the current study captures 98.9% of all echo-based diagnoses as compared to 65.6% by rECHOmmend”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Low left ventricular wall thickness is a highly prevalent label but subjective to high interobserver variation, thus introducing inherent label noise.

“low left ventricular wall thickness is a highly prevalent label but subjective to high interobserver variation, thus introducing inherent label noise.”

May combine two claims.

Restricting the model to a different arbitrary threshold showed a similar AUROC of 87.7%, providing evidence that performance was not dependent on any specific threshold for defining disease.

“Restricting the model to a different arbitrary threshold showed a similar AUROC of 87.7% providing evidence that performance was not dependent on any specific threshold for defining disease.”

May combine two claims.

Rare conditions such as pulmonary regurgitation are under-represented and are highly correlated with other more prevalent conditions, such as tricuspid regurgitation max velocity, which makes it hard to learn patterns that are exclusive to pulmonary regurgitation.

“Rare conditions such as pulmonary regurgitation are under-represented and are highly correlated with other more prevalent conditions, such as tricuspid regurgitation max velocity (Supplementary Fig. 4), which makes it hard to learn patterns that are exclusive to pulmonary regurgitation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most AI-ECG models and their underlying training data are proprietary, limiting comparisons and broader evaluation of generalizability.

“Most importantly, these models and the underlying training data are typically proprietary, limiting comparisons and broader evaluation of generalizability.”

Probably not a substantial claim: may not pass S4 (based).

The multilabel approach best captures collinearity and correlation across component disease labels.

“which was strengthened by the multilabel approach to best capture collinearity and correlation across component disease labels”

May combine two claims.

AI assistance improved cardiologist predictive accuracy, but the combined performance still lagged that of the AI alone.

“Thus, whereas AI assistance for cardiologists in this task improved predictive accuracy, the combined performance still lagged that of the AI alone.”

Classified as probably not a substantial claim.

The cardiologist survey task focused on using the ECG for SHD detection without taking advantage of other information typically available to a physician in clinical practice.

“This task focused on using the ECG for SHD detection without taking advantage of other information (clinical history, physical exam, other testing data) that would typically be available to a physician in clinical practice.”

May combine two claims.

The AUPRCs presented in this study are based on retrospective data in which patients have both ECGs and echocardiograms, which is a population with a much higher prevalence of SHD, and thus not representative of AUPRCs we would expect to see if this model was used as a screening study among patients with ECGs and no echocardiograms.

“The AUPRCs presented in this study are based on retrospective data in which patients have both ECGs and echocardiograms, which is a population with a much higher prevalence of SHD, and thus not representative of AUPRCs we would expect to see if this model was used as a screening study among patients with ECGs and no echocardiograms.”

• AssumptionRests on an assumption: The underlying prevalence of undiagnosed SHD among patients with ECGs and no echocardiograms is not well known.

May combine two claims.

Deep learning analysis can accurately detect SHDs individually and as a composite, with high accuracy across diverse populations in several clinical contexts and geographically distinct external test sets.

“deep learning analysis can accurately detect SHDs individually and as a composite, with high accuracy across diverse populations in several clinical contexts and geographically distinct external test sets.”

Classified as probably not a substantial claim. May combine two claims.

The digital ECG was accessed as a pdf and the name, date and clinical interpretation was cropped out of the image leaving only the waveform and the ECG measurements; age and sex were added to each ECG.

“The digital ECG was accessed as a pdf and the name, date and clinical interpretation was cropped out of the image leaving only the waveform and the ECG measurements (ventricular rate, pulmonary regurgitation interval, QRS interval, Q wave-to-T wave interval and axis). The age (truncated to greater than 90) and sex were added to each ECG.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For each statistical test, 95% CIs were generated using 1,000 bootstrapped estimates.

“For each statistical test, 95% CIs were generated using 1,000 bootstrapped estimates.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This small trial showed promising results, but larger, more pragmatic interventions will be necessary to determine the potential benefit in excess of usual clinical care for EchoNext.

“This small trial showed promising results, but larger, more pragmatic interventions will be necessary to determine the potential benefit in excess of usual clinical care for EchoNext.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The threshold chosen for this study was 45% and results may have differed if a different cutoff was chosen.

“The threshold chosen for this study was 45% and results may have differed if a different cutoff was chosen.”

Classified as probably not a substantial claim.

Patients included in the development cohort were excluded from this analysis.

“Patients included in the development cohort were excluded from this analysis.”

Classified as probably not a substantial claim.

For the primary analysis, data from all eight hospital campuses were blended and split by patient into training, validation and test sets (64%, 16% and 20%).

“data from all eight hospital campuses were blended and split by patient into training, validation and test sets (64%, 16% and 20%)”

Classified as probably not a substantial claim.

These data were harmonized across each echocardiographic reading system and hospital with a minimum of 100 cases audited per label to confirm the accuracy of each analysis.

“These data were harmonized across each echocardiographic reading system and hospital with a minimum of 100 cases audited per label to confirm the accuracy of each analysis.”

May combine two claims.

The multilabel approach helps to best capture the collinearity and correlation among component disease labels and provides insight on the specific disease(s) that leads to a high-risk prediction result, although not conclusive.

“The multilabel approach helps to best capture the collinearity and correlation among component disease labels and provides insight on the specific disease(s) that leads to a high-risk prediction result, although not conclusive.”

Classified as probably not a substantial claim.

We subsequently collected ECG-echocardiogram pairs acquired at NYP from January to 16 September 2023, as a temporally distinct validation set.

“we subsequently collected ECG–echocardiogram pairs acquired at NYP from January to 16 September 2023, as a temporally distinct validation set.”

Classified as probably not a substantial claim.

After the trial completion, EchoNext model performance was assessed on the 100 patients, none of whom were included in model training, validation or testing.

“After the trial completion, EchoNext model performance was assessed on the 100 patients, none of whom were included in model training, validation or testing.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This retrospective study was conducted with approval of the Columbia University and Weill Cornell Institutional Research Boards with waiver of patient consent.

“This retrospective study was conducted with approval of the Columbia University and Weill Cornell Institutional Research Boards with waiver of patient consent.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Subgroup analyses were performed using subsets of age, sex, race and ethnicity.

“Subgroup analyses were performed using subsets of age, sex, race and ethnicity.”

May combine two claims.

Because we took a multilabel approach to train all labels in one model to best capture collinearity, the model optimization was regularized to minimize overall loss across all labels, performance on certain labels might be compromised to ensure better over performance.

“because we took a multilabel approach to train all labels in one model to best capture collinearity, the model optimization was regularized to minimize overall loss across all labels, performance on certain labels might be compromised to ensure better over performance.”

Classified as probably not a substantial claim. May combine two claims.

In all cases, several ECG-echocardiogram pairs were used in training, but the most recent ECG-echocardiogram pair was selected for each unique patient in the validation and test sets.

“several ECG–echocardiogram pairs were used in training, but the most recent ECG–echocardiogram pair was selected for each unique patient in the validation and test sets.”