CoChat
Sign Up
Cited by 241Cardiac Imaging and DiagnosticsArtificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

Deep convolutional neural networks to predict cardiovascular risk from computed tomography

Open access
Zeleznik, Roman, Foldyna, Borek, Eslami, Parastou, Weiss, Jakob, Alexander, Ivanov, Taron, Jana, Parmar, Chintan, Alvi, Raza M. +15 more·2021·Nature Communications
DOI 10.1038/s41467-021-20966-2PMC PMC7846726
Open this paper in CoChat — free accountOpen PDF

Abstract

Coronary artery calcium is an accurate predictor of cardiovascular events. While it is visible on all computed tomography (CT) scans of the chest, this information is not routinely quantified as it requires expertise, time, and specialized equipment. Here, we show a robust and time-efficient deep learning system to automatically quantify coronary calcium on routine cardiac-gated and non-gated CT. As we evaluate in 20,084 individuals from distinct asymptomatic (Framingham Heart Study, NLST) and stable and acute chest pain (PROMISE, ROMICAT-II) cohorts, the automated score is a strong predictor of cardiovascular events, independent of risk factors (multivariable-adjusted hazard ratios up to 4.3), shows high correlation with manual quantification, and robust test-retest reliability. Our results demonstrate the clinical value of a deep learning system for the automated prediction of cardiovascular events. Implementation into clinical practice would address the unmet need of automating proven imaging biomarkers to guide management and improve population health.

AI snapshot

Written from the paper’s full text.

TL;DR

A deep learning system automatically quantified coronary artery calcium from routine cardiac-gated and non-gated CT and predicted cardiovascular events across four large cohorts. The automated score was strongly associated with events, correlated highly with manual scoring, and showed robust test–retest reliability.

Key findings

  • In 20,084 test individuals from FHS-CT2, NLST, PROMISE, and ROMICAT-II, the automated calcium score was a predictor of cardiovascular events independent of risk factors, with multivariable-adjusted hazard ratios up to 4.3 in NLST and odds ratios up to 57.11 in ROMICAT-II.
  • NLST (n = 14,959; ASCVD death n = 288; median follow-up 6.7 years): adjusted for age, sex, diabetes, hypertension, past heart disease, and stroke, HRs versus very low risk were 1.57 (95% CI 0.96–2.57, P = 0.069) for low, 2.79 (1.70–4.57, P < 0.001) for moderate, and 3.87 (2.45–6.11, P < 0.001) for high.
  • PROMISE (n = 4,021; all-cause mortality, MI, or unstable angina n = 130; median 25 months): adjusted for Framingham Risk Score, HRs were 1.90 (1.15–3.15, P = 0.012), 2.57 (1.47–4.50, P = 0.001), and 2.95 (1.71–5.08, P < 0.001) for low, moderate, and high versus very low.
  • ROMICAT-II (n = 441; ACS n = 38): adjusted for TIMI risk score, ORs were 7.70 (2.44–24.30, P = 0.001), 16.20 (4.44–59.13, P < 0.001), and 57.11 (16.55–197.12, P < 0.001) for low, moderate, and high versus very low. Automated scores correlated with manual scores in 5,521 patients (Spearman 0.92, P < 0.0001), combined predicted AUC was 0.74 versus manual 0.75 (P = 0.544), and test–retest ICC was 0.993 versus manual 0.997 in 252 pairs.

Methods

  • Retrospective secondary analysis of a longitudinal primary prevention cohort (FHS-CT1 and FHS-CT2) and three randomized trials (NLST, PROMISE, ROMICAT-II). The system was trained and tuned on 1,636 FHS-CT1 CT scans and tested on 20,084 scans: FHS-CT2 n = 663, NLST n = 14,959, PROMISE n = 4,021, ROMICAT-II n = 441.
  • The deep learning system used four consecutive U-Net-based steps: heart localization, heart segmentation, coronary calcium segmentation, and Agatston calcium score calculation. Scores were stratified as very low (0), low (1–100), moderate (101–300), and high (>300).
  • Analyses included Cox proportional hazards and log-rank tests for NLST and PROMISE, odds ratios for ROMICAT-II, Spearman correlation, ICC, Cohen’s kappa, concordance, and random-effects meta-analysis of AUCs. Manual calcium scores were available for 5,521 subjects.

Limitations

  • The paper states that most participants were non-Hispanic white: FHS 100%, NLST 91%, PROMISE 77%, and ROMICAT-II 66%; generalizability to other demographics requires future study.
  • The paper states that the system evaluated CAC on non-contrast cardiac and chest CT and therefore could not detect noncalcified plaque, which can be present even with a calcium score of zero.
  • Evident from the text: manual reference scores were available for only 5,521 of 20,084 test subjects, and NLST manual validation used a randomly selected subset of 396; FHS-CT2 testing included all CTs, including those initially considered non-diagnostic, while training used only diagnostic-quality CTs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-021-20966-2

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

40 of 57 cited sources checked · 16 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    A dedicated coronary calcium scoring CT is not yet covered by Medicare and most US insurance companies.

    “Despite these recommendations, a dedicated coronary calcium scoring CT is not yet covered by Medicare and most US insurance companies, and for this reason, there is a great deal of interest in deriving the calcium score from routine chest CTs, which are far more common8,28,29.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [8] Hecht, H. S. et al. 2016 SCCT/STR guidelines for coronary artery calcium scoring of noncontrast noncardiac chest CT scans: a report of the Society of Cardiovascular Computed Tomography and Society of Thoracic Radiology. J. Thorac. Imaging 32, W54–W66 (2017). source
    • cites [28] Hecht, H. S. Coronary artery calcium analysis and reporting on noncontrast chest CT scans: a paradigm shift in prevention. Curr. Cardiovasc. Imaging Reports 9 (2016). source
    • cites [29] Lu, M. T. et al. Lung cancer screening eligibility in the community: cardiovascular risk factors, coronary artery calcification, and cardiovascular events. Circulation 134, 897–899 (2016). source

      The paper discusses CMS coverage of lung cancer screening CT and the use of those scans to assess coronary calcium (main text; page unavailable). It does not address whether Medicare or most US insurers cover dedicated coronary calcium scoring CT; searches for that coverage claim found no passage. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
  • Premisecitation

    Persons with a calcium score of zero are at very low risk.

    “Persons with a calcium score of zero are at very low risk24, with increasing risk in the ordinal calcium score tiers identified by the deep learning system25–27.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [24] Blaha, M. J. et al. Role of coronary artery calcium score of zero and other negative risk markers for cardiovascular disease: the multi-ethnic study of atherosclerosis (MESA). Circulation 133, 849–858 (2016). source
  • Findingown data

    In PROMISE, there was a graded association between extent of deep learning calcium score and cardiac events, defined as the composite of death, myocardial infarction, or hospitalization for unstable angina over median 25 months (P < 0.001).

    “In 4021 patients acquired at 193 North American sites, there was a graded association between extent of deep learning calcium score and cardiac events, defined as the composite of death, myocardial infarction, or hospitalization for unstable angina over median 25 months (P < 0.001)(Fig. 2b)19.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [19] Douglas, P. S. et al. PROspective multicenter imaging study for evaluation of chest pain: rationale and design of the PROMISE trial. Am. Heart J. 167, 796–803.e1 (2014). source

      The Abstract Results reports enrollment at 193 US and Canadian sites, and the Abstract Methods identifies death, myocardial infarction, and hospitalization for unstable angina as components of the planned endpoint. But this rationale-and-design paper does not report an analysis of 4,021 patients, deep-learning calcium scores, a graded association with events, median 25-month follow-up, or P < 0.001. Its planned primary endpoint also includes major procedural complications, which the cited sentence omits. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation

    Coronary artery calcium is an accurate predictor of cardiovascular events.

    “Coronary artery calcium is an accurate predictor of cardiovascular events.”
    • Also stated in: § Introduction · boosted
      “One of the strongest known predictors for adverse cardiovascular events is coronary artery calcification, which can be quantified on computed tomography (CT)5,6.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [5] Agatston, A. S. et al. Quantification of coronary artery calcium using ultrafast computed tomography. J. Am. Coll. Cardiol. 15, 827–832 (1990). source

      The Abstract reports that coronary calcium scores distinguished subjects with and without clinical coronary artery disease and gives predictive values for that diagnosis. It does not report whether calcium predicts subsequent cardiovascular events, which is the outcome claimed. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [6] Thanassoulis, G. et al. A genetic risk score is associated with incident cardiovascular disease and coronary artery calcium: the Framingham Heart Study. Circ. Cardiovasc. Genet. 5, 113–121 (2012). source

      The Methods sections on coronary artery calcium measurements and statistical analysis describe measuring CAC and analysing it as an outcome. They do not assess how accurately CAC predicts subsequent cardiovascular events. Searches for CAC’s predictive value for future events found no passage in our copy establishing that claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Premisecitation

    There is increasing risk in the ordinal calcium score tiers identified by the deep learning system.

    “Persons with a calcium score of zero are at very low risk24, with increasing risk in the ordinal calcium score tiers identified by the deep learning system25–27.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [25] Hoffmann, U. et al. Cardiovascular event prediction and risk reclassification by coronary, aortic, and valvular calcification in the Framingham Heart Study. J. Am. Heart Assoc. 5, e003144 (2016). source
    • cites [26] Budoff, M. J. et al. Ten-year association of coronary artery calcium with atherosclerotic cardiovascular disease (ASCVD) events: the multi-ethnic study of atherosclerosis (MESA). Eur. Heart J. 39, 2401–2408 (2018). source
    • cites [27] Mitchell, J. D., Paisley, R., Moon, P., Novak, E. & Villines, T. C. Coronary artery calcium and long-term risk of death, myocardial iInfarction, and stroke: the Walter Reed Cohort Study. JACC Cardiovasc. Imaging 11, 1799–1806 (2018). source
  • May combine two claims. 15
  • Classified as probably not a substantial claim. 30

  • May combine two claims.

    Interpretationcitation

    Risk prediction was robust across multiple clinical scenarios, including a primary prevention asymptomatic setting with non-gated chest CT, and dedicated ECG-gated cardiac CT in stable and acute chest pain settings.

    “Risk prediction was robust across multiple clinical scenarios, including a primary prevention asymptomatic setting with non-gated chest CT (NLST)18, as well as dedicated ECG-gated cardiac CT in stable (PROMISE)19 and acute (ROMICAT-II)20 chest pain setting.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [18] National Lung Screening Trial Research Team et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N. Engl. J. Med. 365, 395–409 (2011). source
    • cites [19] Douglas, P. S. et al. PROspective multicenter imaging study for evaluation of chest pain: rationale and design of the PROMISE trial. Am. Heart J. 167, 796–803.e1 (2014). source
    • cites [20] Hoffmann, U. et al. Design of the rule out myocardial ischemia/infarction using computer assisted tomography: a multicenter randomized comparative effectiveness trial of cardiac computed tomography versus alternative triage strategies in patients with acute chest pain in the emergency department. Am. Heart J. 163, 330–338, 338.e1 (2012). source

      The Patient population and Diagnostic testing sections identify ROMICAT II as an acute chest-pain trial using ECG-gated cardiac CT. But the Conclusion says the trial’s results were still expected; this design paper does not establish that risk prediction was robust in that setting. The NLST and PROMISE settings are attached to other citations, not this one. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Showing patients their coronary calcium provides a teachable moment to empower informed decisions and improve long-term compliance for preventative therapy and lifestyle changes including smoking cessation.

    “Additionally, showing patients their coronary calcium provides a “teachable moment” to empower them to make informed, individualized decisions, and to improve long-term compliance for preventative therapy and lifestyle changes including smoking cessation11,12.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [11] Ravenel, J. G. & Nance, J. W. Coronary artery calcification in lung cancer screening. Transl. Lung Cancer Res. 7, 361–367 (2018). source

      The Introduction describes coronary-risk counselling as a teachable moment that can lead to lifestyle changes or improved pharmacological management. The Conclusions and recommendations also call for discussing CT findings with patients. Neither passage establishes that showing patients their calcium improves long-term adherence to preventive therapy or smoking cessation. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [12] Gupta, A. et al. The identification of calcified coronary plaque is associated with initiation and continuation of pharmacological and lifestyle preventive therapies: a systematic review and meta-analysis. JACC Cardiovasc. Imaging 10, 833–842 (2017). source

      The Results (Characteristics of Included Studies; page not provided) say some studies showed patients their CAC images while explaining their heart-disease risk. The Results (Lifestyle Interventions and Continuation of Preventive Pharmacological Therapy; pages not provided) report greater exercise and dietary change and greater continuation of lipid-lowering medication with nonzero CAC, but no significant effect on smoking cessation. The paper does not establish that showing images itself improved long-term compliance or empowered individualized decisions. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The CT coronary calcium score is a measure of the burden of coronary atherosclerosis and one of the most widely accepted measures of cardiovascular risk.

    “The CT coronary calcium score is a measure of the burden of coronary atherosclerosis and is one of the most widely accepted measures of cardiovascular risk5,6.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [5] Agatston, A. S. et al. Quantification of coronary artery calcium using ultrafast computed tomography. J. Am. Coll. Cardiol. 15, 827–832 (1990). source

      The Abstract describes a CT calcium score that quantifies coronary calcification and reports higher scores in subjects with clinical coronary artery disease, supporting its relevance to coronary disease. It does not establish that the score measures the full burden of coronary atherosclerosis or that it is one of the most widely accepted measures of cardiovascular risk. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [6] Thanassoulis, G. et al. A genetic risk score is associated with incident cardiovascular disease and coronary artery calcium: the Framingham Heart Study. Circ. Cardiovasc. Genet. 5, 113–121 (2012). source

      The Introduction (page unavailable) calls coronary artery calcium a measure of subclinical atherosclerosis, and Methods describes quantifying it by CT. The paper does not establish the further claim that the calcium score is one of the most widely accepted measures of cardiovascular risk. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Deep learning can successfully be used for identifying and segmenting objects within the 3-dimensional image space.

    “Especially in medical imaging there is a large potential as deep learning can successfully be used for identifying and segmenting objects within the 3-dimensional (3D) image space13–16.”
    Quote in the paperSubstantial claim4 of 4 sources checked
    • cites [13] Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. J. W. L. Artificial intelligence in radiology. Nat. Rev. Cancer 18, 500–510 (2018). source
    • cites [14] De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342–1350 (2018). source
    • cites [15] Hosny, A. et al. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Med. 15, e1002711 (2018). source

      The Introduction (page unavailable) says CNNs have shown potential in detection and segmentation, but notes that most work addressed 2D problems and only a few studies used entire 3D tumor volumes. This study’s Methods and Results (pages unavailable) describe manually contoured tumors and a 3D network trained to predict survival, not to identify or segment objects in 3D. The source therefore supports the general imaging applications, but not the specific 3D identification-and-segmentation claim. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [16] Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017). source

  • May combine two claims.

    Interpretationreasoning

    As our deep learning system does not require human input, this makes it an 'end-to-end' solution for accurate and time-efficient cardiovascular risk assessment in clinical settings.

    “As our deep learning system does not require human input, this makes it an ‘end-to-end’ solution for accurate and time-efficient cardiovascular risk assessment in clinical settings32,33.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [32] Takx, R. A. P. et al. Quantification of coronary artery calcium in nongated CT to predict cardiovascular events in male lung cancer screening participants: results of the NELSON study. J. Cardiovasc. Comput. Tomogr. 9, 50–57 (2015). source

      The Conclusions report that CT-derived calcium scores accurately predicted cardiovascular events in male smokers without a CVD history, supporting the risk-assessment rationale. The paper does not establish time efficiency or clinical performance for the citing authors’ deep learning system. Its own automated measurements were visually checked and manually corrected when needed (Imaging predictors, §2.3). (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [33] Chiles, C. et al. Association of coronary artery calcification and mortality in the national lung screening trial: a comparison of three scoring methods. Radiology 276, 82 (2015). source

      The Discussion (page not provided) reports that visual CAC assessment stratified cardiac risk and was faster and simpler than the other methods studied. But the method required radiologists; this paper does not establish that a deep learning system needs no human input or provides an end-to-end clinical solution. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • Rests on an assumption: Deep learning can automate complex assessments that previously could only be done by radiologists, at a scale with higher speed and lower cost.

  • Classified as probably not a substantial claim.

    Premisecitation

    There is great interest in deriving the calcium score from routine chest CTs, which are far more common.

    “Despite these recommendations, a dedicated coronary calcium scoring CT is not yet covered by Medicare and most US insurance companies, and for this reason, there is a great deal of interest in deriving the calcium score from routine chest CTs, which are far more common8,28,29.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [8] Hecht, H. S. et al. 2016 SCCT/STR guidelines for coronary artery calcium scoring of noncontrast noncardiac chest CT scans: a report of the Society of Cardiovascular Computed Tomography and Society of Thoracic Radiology. J. Thorac. Imaging 32, W54–W66 (2017). source
    • cites [28] Hecht, H. S. Coronary artery calcium analysis and reporting on noncontrast chest CT scans: a paradigm shift in prevention. Curr. Cardiovasc. Imaging Reports 9 (2016). source
    • cites [29] Lu, M. T. et al. Lung cancer screening eligibility in the community: cardiovascular risk factors, coronary artery calcification, and cardiovascular events. Circulation 134, 897–899 (2016). source

      In the main text (page unavailable), the paper supports assessing and reporting coronary artery calcium on lung cancer screening CT. It does not say that routine chest CTs are far more common, or that their frequency is the reason for interest in deriving calcium scores from them. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In symptomatic patients, the presence of coronary calcium is associated with future cardiovascular events in the stable chest pain setting.

    “In symptomatic patients, the presence of coronary calcium is associated with future cardiovascular events in the stable chest pain setting9”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Patel, M. R. et al. Correction to: ACC/AATS/AHA/ASE/ASNC/SCAI/SCCT/STS 2017 appropriate use criteria for coronary revascularization in patients with stable ischemic heart disease. J. Nucl. Cardiol. 25, 2191–2192 (2018). source

      The correction (pp. 2191–2192) addresses iFR, FFR, and coronary pressure measurements, not coronary calcium. Searches for calcium and future cardiovascular events in symptomatic patients with stable chest pain found no passage addressing the claimed association. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    The majority of FHS (100%), NLST (91%), PROMISE (77%), and ROMICAT-II (66%) participants were non-Hispanic whites.

    “The majority of FHS (100%)44, NLST (91%)18, PROMISE (77%)45, and ROMICAT-II (66%)46 participants were non-Hispanic whites.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [44] Tsao, C. W. & Vasan, R. S. Cohort profile: the Framingham Heart Study (FHS): overview of milestones in cardiovascular epidemiology. Int. J. Epidemiol. 44, 1800–1813 (2015). source

      In “Description of the FHS” (Table 1; page not provided), the paper lists 100% European ancestry for the Original and Offspring cohorts, not for FHS participants as a whole. The same section describes the racially and ethnically diverse OMNI cohorts, while “Strengths and weaknesses” characterizes FHS participants as predominantly White, with an exception for OMNI. The paper therefore supports a White majority but not the claim that 100% of all FHS participants were non-Hispanic White. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
    • cites [18] National Lung Screening Trial Research Team et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N. Engl. J. Med. 365, 395–409 (2011). source

      The Results section, “Characteristics of the Participants” (page unavailable), refers to demographic characteristics in Table 1, but the passages in our copy do not give the proportion of NLST participants who were non-Hispanic white. Searches for race, ethnicity, and 91% did not locate that figure. The other trials have separate citations and are not attributed to this source. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites [45] Douglas, P. S. et al. Outcomes of anatomical versus functional testing for coronary artery disease. N. Engl. J. Med. 372, 1291–1300 (2015). source

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The CAC score was measured manually by expert readers using the method by Agatston and Janowitz on dedicated workstations, as reported in the parent FHS, PROMISE, and ROMICAT-II studies.

    “The CAC score was measured manually by expert readers using the method by Agatston and Janowitz5,6 on dedicated workstations, as reported in the parent FHS17, PROMISE19, and ROMICAT-II20 studies.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [5] Agatston, A. S. et al. Quantification of coronary artery calcium using ultrafast computed tomography. J. Am. Coll. Cardiol. 15, 827–832 (1990). source
    • cites [6] Thanassoulis, G. et al. A genetic risk score is associated with incident cardiovascular disease and coronary artery calcium: the Framingham Heart Study. Circ. Cardiovasc. Genet. 5, 113–121 (2012). source
    • cites [17] D’Agostino, R. B. et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 117, 743–753 (2008). source

      The Measurement of CVD Risk Factors section describes blood pressure, cholesterol, smoking, and diabetes measurements, but not CAC scoring. Searches for Agatston and Janowitz, expert readers, dedicated workstations, PROMISE, and ROMICAT-II found no passage addressing the claimed measurement method. Page numbers were not available for these passages. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    In NLST, the coronary calcium score was measured manually in 396 randomly selected participants using 3D Slicer (V4).

    “In NLST18, the coronary calcium score was measured manually in 396 randomly selected participants using 3D Slicer (V4).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [18] National Lung Screening Trial Research Team et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N. Engl. J. Med. 365, 395–409 (2011). source

      The Methods > Screening passages describe NLST lung-cancer screening and CT image interpretation, but do not address coronary calcium scoring. Searches for “396,” “coronary calcium score,” and “3D Slicer (V4)” found no passage documenting the claimed participant selection, manual measurement, or software. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    We share our deep learning system, including the trained models, with the community, without restrictions.

    “We also share our deep learning system, including the trained models, with the community, without restrictions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The system calculates the calcium score in under 2 s, without human input.

    “The system calculates the calcium score in under 2 s, without human input.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The system could analyze each image on an average of 1.938 s per scan on a graphics processing unit (GPU) system.

    “The system could analyze each image on an average of 1.938 s per scan on a graphics processing unit (GPU) system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Extensive testing of several cube sizes showed the chosen size worked best as larger cubes increased the training time while the accuracy stayed the same.

    “Extensive testing of several cube sizes showed the chosen size worked best as larger cubes increased the training time while the accuracy stayed the same.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In a few instances, inaccurate heart segmentation led to coronary calcium being outside the ROI of the calcium segmentation network and hence being missed.

    “In a few instances, inaccurate heart segmentation led to coronary calcium being outside the ROI of the calcium segmentation network and hence being missed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    On average, the deep learning system assessed the CAC score in under two seconds per scan.

    “On average, the deep learning system assessed the CAC score in under two seconds per scan.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The deep learning calcium score in 5521 participants had high correlation with human expert readers and demonstrated robust test-retest reliability.

    “The deep learning calcium score in 5521 participants had high correlation with human expert readers and demonstrated robust test–retest reliability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The performance of the deep learning system was tested by reviewing CT scans with discordances between the manual and deep learning coronary calcium categories.

    “The performance of the deep learning system was tested by reviewing CT scans with discordances between the manual and deep learning coronary calcium categories (Supplementary Fig. 7).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    This study was a retrospective secondary analysis of a longitudinal primary prevention cohort (FHS-CT1 and FHS-CT2) and three randomized clinical trials (NLST, PROMISE, ROMICAT-II).

    “This study was a retrospective secondary analysis of a longitudinal primary prevention cohort (FHS-CT1 and FHS-CT2) and three randomized clinical trials (NLST, PROMISE, ROMICAT–II).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Automated quantification of coronary calcium has the potential to improve clinical routine and population health.

    “Automated quantification of coronary calcium has the potential to improve clinical routine and population health.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Coronary calcium, if present, was manually segmented by experienced readers in all subjects.

    “Coronary calcium, if present, was manually segmented by experienced readers in all subjects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    We demonstrate strong robustness of the system by a high correlation with manual scoring in 5521 subjects and high test-retest reliability in data from 252 individuals.

    “Furthermore, we demonstrate strong robustness of the system by a high correlation with manual scoring in 5521 subjects and high test–retest reliability in data from 252 individuals.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    While the FHS-CT1 training cohort included only diagnostic-quality CTs, the FHS-CT2 testing cohort included all CTs including those initially considered non-diagnostic.

    “While the FHS-CT1 training cohort included only diagnostic-quality CTs, the FHS-CT2 testing cohort included all CTs including those initially considered non-diagnostic (Supplementary Fig. 1b).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    We chose to exclude several hundred more CT scans from subjects with no coronary calcium to keep the imbalance between subjects with and without coronary calcium small.

    “Although several hundred more CT scans from subjects with no coronary calcium were available, we chose to exclude them to keep the imbalance between subjects with and without coronary calcium small.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The cohort for training and tuning the three deep learning models consisted of 1636 CT scans: 623 CT scans were from subjects with coronary calcium and 1013 CT scans from subjects with no coronary calcium.

    “The cohort for training and tuning the three deep learning models consisted of 1636 CT scans: 623 CT scans were from subjects with coronary calcium and 1013 CT scans from subjects with no coronary calcium.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In patients with acute chest pain, the presence of coronary calcium is associated with low likelihood of acute coronary syndrome.

    “low likelihood of acute coronary syndrome in patients with acute chest pain10.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Raff, G. L. et al. SCCT guidelines on the use of coronary computed tomographic angiography for patients presenting with acute chest pain to the emergency department: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee. J. Cardiovasc. Comput. Tomogr. 8, 254–271 (2014). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Manually calculated calcium scores from expert readers were available for 5521 subjects and manually segmented hearts for 895 subjects.

    “Manually calculated calcium scores from expert readers were available for 5521 subjects and manually segmented hearts for 895 subjects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Based on the 2018 ACC/AHA guidelines, in persons at intermediate risk (defined as ≥7.5% to <19.9% 10-year risk of cardiovascular events based on risk factors), a calcium score of 0 indicates very low risk and unlikely benefit from statin therapy.

    “Based on the 2018 ACC/AHA guidelines7, in persons at intermediate risk (defined as ≥7.5% to <19.9% 10-year risk of cardiovascular events based on risk factors), a calcium score of 0 indicates very low risk and unlikely benefit from statin therapy”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Grundy, S. M. et al. 2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: executive summary. J. Am. Coll. Cardiol. 73, 3168–3209 (2019). source

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    To verify the results of our deep learning system in the NLST cohort, a subset of randomly chosen 396 subjects were segmented by expert readers.

    “To verify the results of our deep learning system in this cohort, a subset of randomly chosen 396 subjects were segmented by expert readers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    Overall, we included over 20,000 persons drawn from over 200 sites.

    “Overall, we included over 20,000 persons drawn from over 200 sites.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All subjects in the PROMISE and ROMICAT-II testing cohorts were segmented by expert readers.

    “All subjects were segmented by expert readers.”
    • Also stated in: § Study population · plain
      “All subjects were segmented by expert readers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The heart was manually segmented in a subset of 129 randomly selected subjects of the training cohort.

    “Furthermore, the heart was manually segmented in a subset of 129 randomly selected subjects of the training cohort.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Based on the 2018 ACC/AHA guidelines, in persons at intermediate risk, a high calcium score (≥100 or ≥75th centile for age/sex) indicates that a statin should be considered.

    “while a high calcium score (≥100 or ≥75th centile for age/sex) indicates that a statin should be considered7.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Grundy, S. M. et al. 2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: executive summary. J. Am. Coll. Cardiol. 73, 3168–3209 (2019). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The manual calcium score has proven to be an important predictor of cardiovascular events across race and ethnicities.

    “Although the manual calcium score has proven to be an important predictor of cardiovascular events across race and ethnicities, generalizability to other demographics will need to be investigated in future studies47.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] Detrano, R. et al. Coronary calcium as a predictor of coronary events in four racial or ethnic groups. N. Engl. J. Med. 358, 1336–1345 (2008). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Coronary calcium scoring has been recommended by guidelines for risk stratification in primary prevention in asymptomatic individuals.

    “Coronary calcium scoring has been recommended by the guidelines for risk stratification, specifically in the setting of primary prevention in asymptomatic individuals7,8.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] Grundy, S. M. et al. 2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: executive summary. J. Am. Coll. Cardiol. 73, 3168–3209 (2019). source
    • cites [8] Hecht, H. S. et al. 2016 SCCT/STR guidelines for coronary artery calcium scoring of noncontrast noncardiac chest CT scans: a report of the Society of Cardiovascular Computed Tomography and Society of Thoracic Radiology. J. Thorac. Imaging 32, W54–W66 (2017). source

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The benefit of our 3D heart segmentation step is not to find a rectangular ROI, but to narrow the region for coronary calcium segmentation to the heart itself.

    “The benefit of our 3D heart segmentation step is not to find a rectangular ROI, but to narrow the region for coronary calcium segmentation to the heart itself.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    In FHS-CT1, 252 participants underwent cardiac ECG-gated CT twice within 1 h to assess test-retest reliability.

    “In FHS-CT1, 252 participants underwent cardiac ECG-gated CT twice within 1 h to assess test–retest reliability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The deep learning system and the human readers quantified calcium on both scans to assess test-retest reliability.

    “The deep learning system and the human readers quantified calcium on both scans to assess test–retest reliability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We used one unadjusted model, one model adjusted for age and sex, and a third model in which we additionally adjusted for standard cardiovascular risk factors.

    “one unadjusted model, one model adjusted for age and sex, and a third model in which we additionally adjusted for standard cardiovascular risk factors”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    The measurement of coronary calcium score requires radiological expertise, time, and specialized coronary calcium quantification equipment.

    “However, the measurement requires radiological expertise, time, and specialized coronary calcium quantification equipment.”

  • The paper relies on this without support.

    This essentially free available information is usually not reported.

    “As a result, this essentially free available information is usually not reported.”

  • The paper relies on this without support.

    Deep learning can automate complex assessments that previously could only be done by radiologists, at a scale with higher speed and lower cost.

    “A major advantage is that deep learning can automate complex assessments that previously could only be done by radiologists, but now is feasible at a scale with a higher speed and lower cost.”

  • The paper relies on this without support.

    Traditionally, coronary calcium scoring requires special software, manual measurement by trained experts and dedicated ECG-gated cardiac CT.

    “Traditionally, coronary calcium scoring requires special software, manual measurement by trained experts and dedicated ECG-gated cardiac CT.”

  • The paper relies on this without support.

    As a consequence, the calcium score is often not reported on routine noncardiac chest CT.

    “As a consequence, the calcium score is often not reported on routine noncardiac chest CT”

  • The authors state this as an assumption.

    Before clinical introduction of automated systems for cardiovascular risk prediction, their generalizability across multiple clinical scenarios and institutions needs to be demonstrated.

    “This is essential, as before clinical introduction can be considered, the generalizability of these automated systems needs to be demonstrated as they need to be able to predict cardiovascular events of asymptomatic individuals across multiple clinical scenarios and work robustly on data from multiple institutions.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Discussion count omits the FHS-CT2 cohort

    Minor

    The four test-cohort sizes in Table 1 sum to 20,084, not 19,421. The stated 19,421 equals the total without the 663 FHS-CT2 participants, despite the sentence referring to four studies.

    Reported 19421 · recomputed 20084

  • ROMICAT-II ACS endpoint is given as both 28 and 30 days

    Minor

    The results describe the ROMICAT-II association at 28 days, but the caption for the cited figure calls its ACS rate a 30-day rate.

  • Statistical methods describe quartiles instead of the reported CAC groups

    Minor

    The methods say the regression compares CAC quartiles, while Table 2 defines and reports groups using fixed CAC-score ranges, not quartiles.

  • ROMICAT-II effects are called hazard ratios but tabulated as odds ratios

    Minor

    The results describe the ROMICAT-II risk estimates as HRs, whereas Table 2 labels the corresponding estimates as ORs.

4 of 4

Cites

Flagged first, then by citations

1 of the 59 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Zeleznik, R., Foldyna, B., Eslami, P., Weiss, J., Alexander, I., Taron, J., Parmar, C., Alvi, R. M., Banerji, D., Uno, M., Kikuchi, Y., Karady, J., Zhang, L., Scholtz, J.-E., Mayrhofer, T., Lyass, A., Mahoney, T. F., Massaro, J. M., Vasan, R. S., Douglas, P. S., Hoffmann, U., Lu, M. T., & Aerts, H. J. W. L.
Affiliations
  • Brigham and Women's Hospital (US)
  • Harvard University (US)
  • Boston University (US)
  • Duke University (US)
Corresponding author
Aerts, Hugo J. W. L.
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2021-01-29
Volume
12
Issue
1
Pages
715-715
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-021-20966-2
PMC
PMC7846726
OpenAlex
W3126008863
Cited by
240
References
59
Citations by year
2026: 33 · 2025: 48 · 2024: 61 · 2023: 36 · 2022: 42 · 2021: 20
FWCI
19.23
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (submittedVersion, other-oa)
  • PubMed Central (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Funders
  • American Heart Association (5T32HL076136, 5K24HL113128, 18UNPG34030172, 5R01-HL109711, HHSN268201500001I, 5U01HL123339)
  • American College of Radiology Imaging Network
  • European Commission
  • Deutsche Forschungsgemeinschaft
  • National Institutes of Health (75N92019D00031, NIH-USA R35CA22052, 5k24hl113128-09, 5u01hl123339-06, R35CA22052, n01hc25195-41-0-1, 18UNPG34030172, HHSN268201500001I, U01-CA209414, NIH/NHLBI 5K24HL113128, NIH‐USA U24CA194354, NIH-USA U01CA190234, NIH-USA U01CA209414, U24CA194354, NHLBI 5K24HL113128, NO1-HC-25195, NIH-USA U24CA194354, NIH-USA U01CA190234, U01CA190234, 5K24HL113128)
  • National Heart, Lung, and Blood Institute (K24HL113128, 5K24HL113128, HHSN268201500001I, 5U01HL123339, HHSN268201500001C, 75N92019D00031, N01-HC-025195, R01HL109711, U01 CA209414, T32HL076136, NO1‐HC‐25195, 5R01-HL109711, U01 HL123339, E0583118)
  • National Cancer Institute (U24-CA194354, U01-CA190234, U01-CA209414, R35CA22052)
  • H2020 European Research Council (866504)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Cardiac Imaging and Diagnostics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
coronary artery calcium, cardiovascular risk prediction, deep learning, computed tomography, cardiovascular events, imaging biomarkers, test-retest reliability, chest pain, Framingham Heart Study, promise
MeSH terms
Deep Learning, Heart Disease Risk Factors, Aged, Calcium, Cardiovascular Diseases, Chest Pain, Coronary Vessels, Female, Follow-Up Studies, Humans, Image Processing, Computer-Assisted, Male, Middle Aged, Retrospective Studies, Tomography, X-Ray Computed, Reproducibility of Results, Risk Assessment, Asymptomatic Diseases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review