Proposed Requirements for Cardiovascular Imaging-Related Machine Learning Evaluation (PRIME): A Checklist
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Machine Learning (ML) has been increasingly used within cardiology, particularly in the domain of cardiovascular imaging. Due to the inherent complexity and flexibility of ML algorithms, inconsistencies in the model performance and interpretation may occur. Several review articles have been recently published that introduce the fundamental principles and clinical application of ML for cardiologists. This paper builds on these introductory principles and outlines a more comprehensive list of crucial responsibilities that need to be completed when developing ML models. This paper aims to serve as a scientific foundation to aid investigators, data scientists, authors, editors, and reviewers involved in machine learning research with the intent of uniform reporting of ML investigations. An independent multidisciplinary panel of ML experts, clinicians, and statisticians worked together to review the theoretical rationale underlying 7 sets of requirements that may reduce algorithmic errors and biases. Finally, the paper summarizes a list of reporting items as an itemized checklist that highlights steps for ensuring correct application of ML models and the consistent reporting of model specifications and results. It is expected that the rapid pace of research and development and the increased availability of real-world evidence may require periodic updates to the checklist.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · cited 1×
“, local binary patterns) (38) and scale-invariant feature transform (39); 2) classical ML methods for dimensionality reduction (e.”
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
A Global Geometric Framework for Nonlinear Dimensionality Reduction
Tenenbaum, J. B., Silva, V. D., & Langford, J. C. (2000) · Science · doi:10.1126/science.290.5500.2319 · Cited by 13,585
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 2×
“This process is repeated multiple times to estimate the learning method’s generalization performance (Figure 6 II) (33,76).”
Self-citationShin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Cited by
Ethical and Bias Considerations in Artificial Intelligence/Machine Learning
Hanna, M. G., Pantanowitz, L., Jackson, B., Palmer, O., Visweswaran, S., Pantanowitz, J., Deebajah, M., & Rashidi, H. H. (2025) · Modern Pathology · doi:10.1016/j.modpat.2024.100686 · Cited by 535
Machine Learning and the Future of Cardiovascular Care
Quer, G., Arnaout, R., Henne, M., & Arnaout, R. (2021) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.11.030 · Cited by 337
Self-citationChoi, A. D., Marques, H., Kumar, V., Griffin, W. F., Rahban, H., Karlsberg, R. P., Zeman, R. K., Katz, R. J., & Earls, J. P. (2021) · Journal of Cardiovascular Computed Tomography · doi:10.1016/j.jcct.2021.05.004 · Cited by 231
Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis
Kolbinger, F. R., Veldhuizen, G. P., Zhu, J., Truhn, D., & Kather, J. N. (2024) · Communications Medicine · doi:10.1038/s43856-024-00492-0 · Cited by 160
Cai, Y., Cai, Y.-Q., Tang, L.-Y., Wang, Y.-H., Gong, M., Jing, T.-C., Li, H.-J., Li-Ling, J., Hu, W., Yin, Z., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03273-7 · Cited by 122
Artificial Intelligence in Cardiovascular Care—Part 2: Applications
Jain, S. S., Elias, P., Poterucha, T., Randazzo, M., Lopez Jimenez, F., Khera, R., Perez, M., Ouyang, D., Pirruccello, J., Salerno, M., et al. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.03.401 · Cited by 74
Self-citationMust-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning
Koçak, B., Cuocolo, R., Santos, D. P. D., Stanzione, A., & Ugga, L. (2023) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2022.2022-11-51 · Cited by 64
Kuroda, Y., Kaneko, T., Yoshikawa, H., Uchiyama, S., Nagata, Y., Matsushita, Y., Hiki, M., Minamino, T., Takahashi, K., Daida, H., et al. (2023) · PLOS ONE · doi:10.1371/journal.pone.0281127 · Cited by 36
Self-citationAdvances in Machine Learning Approaches to Heart Failure with Preserved Ejection Fraction
Ahmad, F. S., Luo, Y., Wehbe, R. M., Thomas, J. D., & Shah, S. J. (2022) · Heart Failure Clinics · doi:10.1016/j.hfc.2021.12.002 · Cited by 34
AI-Defined Cardiac Anatomy Improves Risk Stratification of Hybrid Perfusion Imaging
Miller, R. J. H., Shanbhag, A., Killekar, A., Lemley, M., Bednarski, B., Kavanagh, P. B., Feher, A., Miller, E. J., Bateman, T., Builoff, V., et al. (2024) · JACC: Cardiovascular Imaging · doi:10.1016/j.jcmg.2024.01.006 · Cited by 33
Self-citationMitigating bias in clinical machine learning models
Perez-Downes, J. C., Tseng, A. S., McConn, K. A., Elattar, S. M., Sokumbi, O., Sebro, R. A., Allyse, M. A., Dangott, B. J., Carter, R. E., & Adedinsewo, D. (2024) · Current Treatment Options in Cardiovascular Medicine · doi:10.1007/s11936-023-01032-0 · Cited by 23
Koçak, B., Keleş, A., & Köse, F. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.232604 · Cited by 12
Artificial Intelligence in Coronary Artery Calcium Scoring
Aromiwura, A. A., & Kalra, D. K. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13123453 · Cited by 11
“…a certain degree of false positive rates is expected with imaging tests, it is essential to minimize this in the higher CAC risk groups (>100 AU) where results significantly impact medical decision-making, to avoid over-utilization of resources for primary prevention and further diagnostic testing.”
Background
Metadata
- Authors
- Sengupta, P. P., Shrestha, S., Berthon, B., Messas, E., Donal, E., Tison, G. H., Min, J. K., D’hooge, J., Voigt, J.-U., Dudley, J., Verjans, J. W., Shameer, K., Johnson, K., Lovstakken, L., Tabassian, M., Piccirilli, M., Pernot, M., Yanamala, N., Duchateau, N., Kagiyama, N., Bernard, O., Slomka, P., Deo, R., & Arnaout, R.
- Venue
- JACC: Cardiovascular Imaging
- Publisher
- Elsevier BV
- Published
- 2020-09
- Volume
- 13
- Issue
- 9
- Pages
- 2017-2035
- ISSN
- 1876-7591 (ISSN-L)
- 1936-878X
- OpenAlex
- W3083342460
- Cited by
- 221
- References
- 31
- FWCI
- 7.03
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.