Cited by 244Cardiac Imaging and DiagnosticsArtificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

Deep Learning for Automatic Calcium Scoring in CT: Validation Using Multiple Cardiac CT and Chest CT Protocols

Open access
van Velzen, Sanne G. M., Lessmann, Nikolas, Velthuis, Birgitta K., Bank, Ingrid E. M., van den Bongard, Desiree H. J. G., Leiner, Tim, de Jong, Pim A., Veldhuis, Wouter B. +6 more·2020·Radiology

Abstract

Background Although several deep learning (DL) calcium scoring methods have achieved excellent performance for specific CT protocols, their performance in a range of CT examination types is unknown. Purpose To evaluate the performance of a DL method for automatic calcium scoring across a wide range of CT examination types and to investigate whether the method can adapt to different types of CT examinations when representative images are added to the existing training data set. Materials and Methods The study included 7240 participants who underwent various types of nonenhanced CT examinations that included the heart: coronary artery calcium (CAC) scoring CT, diagnostic CT of the chest, PET attenuation correction CT, radiation therapy treatment planning CT, CAC screening CT, and low-dose CT of the chest. CAC and thoracic aorta calcification (TAC) were quantified using a convolutional neural network trained with (a) 1181 low-dose chest CT examinations (baseline), (b) a small set of examinations of the respective type supplemented to the baseline (data specific), and (c) a combination of examinations of all available types (combined). Supplemental training sets contained 199–568 CT images depending on the calcium burden of each population. The DL algorithm performance was evaluated with intraclass correlation coefficients (ICCs) between DL and manual (Agatston) CAC and (volume) TAC scoring and with linearly weighted κ values for cardiovascular risk categories (Agatston score; cardiovascular disease risk categories: 0, 1–10, 11–100, 101–400, >400). Results At baseline, the DL algorithm yielded ICCs of 0.79–0.97 for CAC and 0.66–0.98 for TAC across the range of different types of CT examinations. ICCs improved to 0.84–0.99 (CAC) and 0.92–0.99 (TAC) for CT protocol–specific training and to 0.85–0.99 (CAC) and 0.96–0.99 (TAC) for combined training. For assignment of cardiovascular disease risk category, the κ value for all test CT scans was 0.90 (95% confidence interval [CI]: 0.89, 0.91) for the baseline training. It increased to 0.92 (95% CI: 0.91, 0.93) for both data-specific and combined training. Conclusion A deep learning calcium scoring algorithm for quantification of coronary and thoracic calcium was robust, despite substantial differences in CT protocol and variations in subject population. Augmenting the algorithm training with CT protocol–specific images further improved algorithm performance. © RSNA, 2020 See also the editorial by Vannier in this issue.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1148/radiol.2020191621

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A combined solution capable of analyzing cardiac and non-gated chest CT as presented in our investigation has only been proposed by de Vos et al. and van Velzen et al.

    “A combined solution capable of analyzing cardiac and non-gated chest CT as presented in our investigation has only been proposed by de Vos et al.36 and van Velzen et al.41.”

    From Deep convolutional neural networks to predict cardiovascular risk from computed tomography (2021) · cites this paper as [41]

    The title in our indexed copy identifies van Velzen et al. (Radiology, 2020, pp. 66–79) as a study of automatic calcium scoring across cardiac CT and chest CT protocols. The available passages do not specify that the chest CT was non-gated or establish the sentence’s “only” claim; searches for non-gated CT and a combined method found no further substantive passage. (LLM, conf=0.75)

    Experimental — accuracy not yet measured
  • Other studies that investigated deep learning algorithms for automated coronary calcium quantification used smaller cohorts or proprietary technologies.

    “Although other studies have investigated deep learning algorithms for automated coronary calcium quantification34–42, they used smaller cohorts or proprietary technologies.”

    From Deep convolutional neural networks to predict cardiovascular risk from computed tomography (2021) · cites this paper as [41]

    The paper’s title and Abstract identify it as a study of deep-learning-based automatic calcium scoring in CT, supporting its inclusion among studies investigating automated coronary calcium quantification. The available passages do not establish that it used a smaller cohort or proprietary technology; searches for those comparisons found no matching passage. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
van Velzen, S. G. M., Lessmann, N., Velthuis, B. K., Bank, I. E. M., van den Bongard, D. H. J. G., Leiner, T., de Jong, P. A., Veldhuis, W. B., Correa, A., Terry, J. G., Carr, J. J., Viergever, M. A., Verkooijen, H. M., & Išgum, I.
Affiliations
Radboud University Medical Center (NL)
Venue
Radiology
Published
2020-04
Volume
295
Issue
1
Pages
66-79
ISSN
  • 0033-8419 (ISSN-L)
  • 1527-1315
OpenAlex
W3005949594
Cited by
244
References
34
Citations by year
2026: 29 · 2025: 26 · 2024: 40 · 2023: 36 · 2022: 48 · 2021: 47 · 2020: 18
FWCI
18.14
Type
journal article
Language
en
Open access
Locations
  • Radiology (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Funders
  • U.S. Department of Health and Human Services (HHSN268201800014I, HHSN268201800010I, HHSN268201800013I, HHSN268201800012I, HHSN268201800015I)
  • Jackson State University (HHSN268201800015I/HHSN26800001, HHSN268201800013 I, HHSN268201800014 I)
  • Mississippi State Department of Health (HHSN26800001, HHSN268201800015 I, HHSN268201800015I/HHSN26800001)
  • National Institutes of Health (HHSN268201800010I, HHSN268201800012I, HHSN268201800014I, HHSN26800001, HHSN268201800013I, HHSN268201800015I)
  • National Heart, Lung, and Blood Institute (HHSN268201800013I, HHSN26800001, HHSN268201800012C, HHSN268201800010 I, HHSN268201800014 I, HHSN268201800012 I, HHSN268201100011I, HHSN268201800014C, HHSN268201800015I, HHSN268201800011C, HHSN268201800011I)
  • National Cancer Institute (HHSN261201800014I, HHSN261201800012I, HHSN261201800010I)
  • National Institute on Minority Health and Health Disparities (HHSN268201800012I, HHSN268201800010I, HHSN268201800014I, HHSN268201800015I, HHSN268201800013I)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Cardiac Imaging and Diagnostics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
coronary artery calcium scoring, thoracic aortic calcification, deep learning, convolutional neural networks, chest CT, low-dose CT, cardiovascular risk stratification, Agatston score, cardiovascular risk prediction, cardiovascular risk prediction models
MeSH terms
Deep Learning, Aged, Clinical Protocols, Coronary Artery Disease, Female, Heart, Humans, Male, Middle Aged, Retrospective Studies, Thorax, Tomography, X-Ray Computed, Vascular Calcification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review