The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Deep Learning for Automatic Calcium Scoring in CT: Validation Using Multiple Cardiac CT and Chest CT Protocols
Open accessAbstract
Background Although several deep learning (DL) calcium scoring methods have achieved excellent performance for specific CT protocols, their performance in a range of CT examination types is unknown. Purpose To evaluate the performance of a DL method for automatic calcium scoring across a wide range of CT examination types and to investigate whether the method can adapt to different types of CT examinations when representative images are added to the existing training data set. Materials and Methods The study included 7240 participants who underwent various types of nonenhanced CT examinations that included the heart: coronary artery calcium (CAC) scoring CT, diagnostic CT of the chest, PET attenuation correction CT, radiation therapy treatment planning CT, CAC screening CT, and low-dose CT of the chest. CAC and thoracic aorta calcification (TAC) were quantified using a convolutional neural network trained with (a) 1181 low-dose chest CT examinations (baseline), (b) a small set of examinations of the respective type supplemented to the baseline (data specific), and (c) a combination of examinations of all available types (combined). Supplemental training sets contained 199–568 CT images depending on the calcium burden of each population. The DL algorithm performance was evaluated with intraclass correlation coefficients (ICCs) between DL and manual (Agatston) CAC and (volume) TAC scoring and with linearly weighted κ values for cardiovascular risk categories (Agatston score; cardiovascular disease risk categories: 0, 1–10, 11–100, 101–400, >400). Results At baseline, the DL algorithm yielded ICCs of 0.79–0.97 for CAC and 0.66–0.98 for TAC across the range of different types of CT examinations. ICCs improved to 0.84–0.99 (CAC) and 0.92–0.99 (TAC) for CT protocol–specific training and to 0.85–0.99 (CAC) and 0.96–0.99 (TAC) for combined training. For assignment of cardiovascular disease risk category, the κ value for all test CT scans was 0.90 (95% confidence interval [CI]: 0.89, 0.91) for the baseline training. It increased to 0.92 (95% CI: 0.91, 0.93) for both data-specific and combined training. Conclusion A deep learning calcium scoring algorithm for quantification of coronary and thoracic calcium was robust, despite substantial differences in CT protocol and variations in subject population. Augmenting the algorithm training with CT protocol–specific images further improved algorithm performance. © RSNA, 2020 See also the editorial by Vannier in this issue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A combined solution capable of analyzing cardiac and non-gated chest CT as presented in our investigation has only been proposed by de Vos et al.36 and van Velzen et al.41.”
From Deep convolutional neural networks to predict cardiovascular risk from computed tomography (2021) · cites this paper as [41]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although other studies have investigated deep learning algorithms for automated coronary calcium quantification34–42, they used smaller cohorts or proprietary technologies.”
From Deep convolutional neural networks to predict cardiovascular risk from computed tomography (2021) · cites this paper as [41]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reduced Lung-Cancer Mortality with Low-Dose Computed Tomographic Screening
The National Lung Screening Trial Research Team (2011) · New England Journal of Medicine · doi:10.1056/nejmoa1102873 · Cited by 11,064
Quantification of coronary artery calcium using ultrafast computed tomography
Agatston, A. S., Janowitz, W. R., Hildner, F. J., Zusmer, N. R., Viamonte, M., & Detrano, R. (1990) · Journal of the American College of Cardiology · doi:10.1016/0735-1097(90)90282-t · Cited by 7,786
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Grundy, S. M., Stone, N. J., Bailey, A. L., Beam, C., Birtcher, K. K., Blumenthal, R. S., Braun, L. T., de Ferranti, S., Faiella-Tommasino, J., Forman, D. E., et al. (2019) · Circulation · doi:10.1161/cir.0000000000000625 · Cited by 6,185
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277
Setio, A. A. A., Traverso, A., de Bel, T., Berens, M. S. N., van den Bogaard, C., Cerello, P., Chen, H., Dou, Q., Fantacci, M. E., Geurts, B. H. J., et al. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.06.015 · Cited by 1,456
Long-Term Prognosis Associated With Coronary Calcification
Budoff, M. J., Shaw, L. J., Liu, S. T., Weinstein, S. R., Tseng, P. H., Flores, F. R., Callister, T. Q., Raggi, P., Berman, D. S., & Mosler, T. P. (2007) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2006.10.079 · Cited by 1,341
Grundy, S. M., Stone, N. J., Bailey, A. L., Beam, C. T., Birtcher, K. K., Blumenthal, R. S., Braun, L. T., de Ferranti, S. D., Faiella-Tommasino, J., Forman, D. E., et al. (2019) · Circulation · doi:10.1161/cir.0000000000000624 · Cited by 933
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Taylor, H. A., WILSON, J. G., Jones, D. W., Sarpong, D. F., Srinivasan, A., Garrison, R. J., Nelson, C. R., & Wyatt, S. B. (2005) · PubMed · Cited by 525
Hecht, H. S., Cronin, P., Blaha, M. J., Budoff, M. J., Kazerooni, E. A., Narula, J., Yankelevitz, D., & Abbara, S. (2017) · Journal of Cardiovascular Computed Tomography · doi:10.1016/j.jcct.2016.11.003 · Cited by 492
Blaha, M. J., Cainzos-Achirica, M., Greenland, P., McEvoy, J. W., Blankstein, R., Budoff, M. J., Dardari, Z., Sibley, C. T., Burke, G. L., Kronmal, R. A., et al. (2016) · Circulation · doi:10.1161/circulationaha.115.018524 · Cited by 485
Carr, J. J., Jacobs, D. R., Terry, J. G., Shay, C. M., Sidney, S., Liu, K., Schreiner, P. J., Lewis, C. E., Shikany, J. M., Reis, J. P., et al. (2017) · JAMA Cardiology · doi:10.1001/jamacardio.2016.5493 · Cited by 361
Self-citationChiles, C., Duan, F., Gladish, G. W., Ravenel, J. G., Baginski, S. G., Snyder, B. S., DeMello, S., Desjardins, S. S., Munden, R. F., & For the NLST Study Team (2015) · Radiology · doi:10.1148/radiol.15142062 · Cited by 356
Shemesh, J., Henschke, C. I., Shaham, D., Yip, R., Farooqi, A. O., Cham, M. D., McCauley, D. I., Chen, M., Smith, J. P., Libby, D. M., et al. (2010) · Radiology · doi:10.1148/radiol.10100383 · Cited by 319
Coronary Artery Calcium Scanning
Hecht, H. S. (2015) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2015.02.006 · Cited by 309
Automatic Calcium Scoring in Low-Dose Chest CT Using Deep Neural Networks With Dilated Convolutions
Lessmann, N., van Ginneken, B., Zreik, M., de Jong, P. A., de Vos, B. D., Viergever, M. A., & Isgum, I. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2769839 · Cited by 263
Self-citationJacobs, P. C. A., Gondrie, M. J. A., van der Graaf, Y., de Koning, H. J., Išgum, I., van Ginneken, B., & Mali, W. P. T. M. (2012) · American Journal of Roentgenology · doi:10.2214/ajr.10.5577 · Cited by 178
Self-citationConvNet-Based Localization of Anatomical Structures in 3-D Medical Images
de Vos, B. D., Wolterink, J. M., de Jong, P. A., Leiner, T., Viergever, M. A., & Isgum, I. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2673121 · Cited by 133
Self-citationAutomated Coronary Artery Calcification Scoring in Non-Gated Chest CT: Agreement and Reliability
Takx, R. A. P., de Jong, P. A., Leiner, T., Oudkerk, M., de Koning, H. J., Mol, C. P., Viergever, M. A., & Išgum, I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0091239 · Cited by 128
Self-citationWolterink, J. M., Leiner, T., Takx, R. A. P., Viergever, M. A., & Išgum, I. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2412651 · Cited by 119
Self-citationIšgum, I., Rutten, A., Prokop, M., & van Ginneken, B. (2007) · Medical Physics · doi:10.1118/1.2710548 · Cited by 106
Self-citationTakasu, J., Katz, R., Nasir, K., Carr, J. J., Wong, N. D., Detrano, R., & Budoff, M. J. (2008) · American Heart Journal · doi:10.1016/j.ahj.2007.11.019 · Cited by 106
Self-citationAutomated Agatston score computation in non-ECG gated CT scans using deep learning
González, G., Washko, G. R., Estépar, R. S. J., Cazorla, M., & Cano-Espinosa, C. (2018) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2293681 · Cited by 87
Vessel Specific Coronary Artery Calcium Scoring
Shahzad, R., van Walsum, T., Schaap, M., Rossi, A., Klein, S., Weustink, A. C., de Feyter, P. J., A, L. J. V. V., & Niessen, W. J. (2012) · Academic Radiology · doi:10.1016/j.acra.2012.07.018 · Cited by 81
Automatic determination of cardiovascular risk by CT attenuation correction maps in Rb-82 PET/CT
Išgum, I., de Vos, B. D., Wolterink, J. M., Dey, D., Berman, D. S., Rubeaux, M., Leiner, T., & Slomka, P. J. (2017) · Journal of Nuclear Cardiology · doi:10.1007/s12350-017-0866-3 · Cited by 66
Self-citationA supervised classification-based method for coronary calcium detection in non-contrast CT
Kurkure, U., Chittajallu, D. R., Brunner, G., Le, Y. H., & Kakadiaris, I. A. (2010) · International journal of cardiac imaging · doi:10.1007/s10554-010-9607-2 · Cited by 62
Mylonas, I., Kazmi, M. H., Fuller, L., deKemp, R. A., Yam, Y., Chen, L., Beanlands, R. S., & Chow, B. J. W. (2012) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jes079 · Cited by 54
Sevrukov, A., Bland, M., & Kondos, G. T. (2005) · American Journal of Roentgenology · doi:10.2214/ajr.04.1589 · Cited by 50
Gernaat, S. A. M., Išgum, I., de Vos, B. D., Takx, R. A. P., Young‐Afat, D. A., Rijnberg, N., Grobbee, D. E., van der Graaf, Y., de Jong, P. A., Leiner, T., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0167925 · Cited by 48
Self-citationAutomated aortic calcium scoring on low-dose chest computed tomography.
Išgum, I., Rutten, A., Prokop, M., Staring, M., Klein, S., Pluim, J., Viergever, M., & van Ginneken, B. (2010) · Medical Physics (Lancaster) · doi:10.1118/1.3284211 · Cited by 44
Self-citationGernaat, S. A. M., van Velzen, S. G. M., Koh, V., Emaus, M. J., Išgum, I., Leßmann, N., Moes, S. L., Jacobson, A., Tan, P. W., Grobbee, D. E., et al. (2018) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2018.04.011 · Cited by 43
Self-citationSung, J. H., Yeboah, J., Lee, J. C., Smith, C. L., Terry, J. G., Sims, M., Samdarshi, T. E., Musani, S. K., Fox, E. R., Ge, Y., et al. (2015) · Journal of Advances in Medicine and Medical Research · doi:10.9734/bjmmr/2016/21449 · Cited by 32
Self-citation
Research works printed but not linked: 4
- Grundy SM, 2019, Circulation, 139, e1046
- Bank IEM, 2017, Ischaemic Heart Disease: Early Recognition and Risk Disparities [dissertation]. Vol. Chapter 3
- Taylor HA, 2005, Ethn Dis, 15, S6
- Cano-Espinosa C, 2018, Proceedings of SPIE: medical imaging 2018—image processing, 10574, 105742K
Cited by
Sorted by citations
Adams, S. J., Stone, E., Baldwin, D. R., Vliegenthart, R., Lee, P., & Fintelmann, F. J. (2023) · The Lancet · doi:10.1016/s0140-6736(22)01694-4 · Cited by 467
Machine Learning and the Future of Cardiovascular Care
Quer, G., Arnaout, R., Henne, M., & Arnaout, R. (2021) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.11.030 · Cited by 353
Deep convolutional neural networks to predict cardiovascular risk from computed tomography
Zeleznik, R., Foldyna, B., Eslami, P., Weiss, J., Alexander, I., Taron, J., Parmar, C., Alvi, R. M., Banerji, D., Uno, M., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-20966-2 · Cited by 240
Automated coronary calcium scoring using deep learning with multicenter external validation
Eng, D. K.-M., Chute, C., Khandwala, N., Rajpurkar, P., Long, J., Shleifer, S., Khalaf, M. H., Sandhu, A. T., Rodríguez, F., Maron, D. J., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00460-1 · Cited by 189
Eche, T., Schwartz, L. H., Mokrane, F.-Z., & Dercle, L. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021210097 · Cited by 163
Metadata
- Authors
- van Velzen, S. G. M., Lessmann, N., Velthuis, B. K., Bank, I. E. M., van den Bongard, D. H. J. G., Leiner, T., de Jong, P. A., Veldhuis, W. B., Correa, A., Terry, J. G., Carr, J. J., Viergever, M. A., Verkooijen, H. M., & Išgum, I.
- Affiliations
- Radboud University Medical Center (NL)
- ORCID
- Venue
- Radiology
- Published
- 2020-04
- Volume
- 295
- Issue
- 1
- Pages
- 66-79
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC7106943
- OpenAlex
- W3005949594
- Cited by
- 244
- References
- 34
- Citations by year
- 2026: 29 · 2025: 26 · 2024: 40 · 2023: 36 · 2022: 48 · 2021: 47 · 2020: 18
- FWCI
- 18.14
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Funders
- U.S. Department of Health and Human Services (HHSN268201800014I, HHSN268201800010I, HHSN268201800013I, HHSN268201800012I, HHSN268201800015I)
- Jackson State University (HHSN268201800015I/HHSN26800001, HHSN268201800013 I, HHSN268201800014 I)
- Mississippi State Department of Health (HHSN26800001, HHSN268201800015 I, HHSN268201800015I/HHSN26800001)
- National Institutes of Health (HHSN268201800010I, HHSN268201800012I, HHSN268201800014I, HHSN26800001, HHSN268201800013I, HHSN268201800015I)
- National Heart, Lung, and Blood Institute (HHSN268201800013I, HHSN26800001, HHSN268201800012C, HHSN268201800010 I, HHSN268201800014 I, HHSN268201800012 I, HHSN268201100011I, HHSN268201800014C, HHSN268201800015I, HHSN268201800011C, HHSN268201800011I)
- National Cancer Institute (HHSN261201800014I, HHSN261201800012I, HHSN261201800010I)
- National Institute on Minority Health and Health Disparities (HHSN268201800012I, HHSN268201800010I, HHSN268201800014I, HHSN268201800015I, HHSN268201800013I)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Cardiac Imaging and Diagnostics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- coronary artery calcium scoring, thoracic aortic calcification, deep learning, convolutional neural networks, chest CT, low-dose CT, cardiovascular risk stratification, Agatston score, cardiovascular risk prediction, cardiovascular risk prediction models
- MeSH terms
- Deep Learning, Aged, Clinical Protocols, Coronary Artery Disease, Female, Heart, Humans, Male, Middle Aged, Retrospective Studies, Thorax, Tomography, X-Ray Computed, Vascular Calcification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
