The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Recommendations for Reporting Machine Learning Analyses in Clinical Research
Open accessAbstract
Use of machine learning (ML) in clinical research is growing steadily given the increasing availability of complex clinical data sets. ML presents important advantages in terms of predictive performance and identifying undiscovered subpopulations of patients with specific physiology and prognoses. Despite this popularity, many clinicians and researchers are not yet familiar with evaluating and interpreting ML analyses. Consequently, readers and peer-reviewers alike may either overestimate or underestimate the validity and credibility of an ML-based model. Conversely, ML experts without clinical experience may present details of the analysis that are too granular for a clinical readership to assess. Overwhelming evidence has shown poor reproducibility and reporting of ML models in clinical research suggesting the need for ML analyses to be presented in a clear, concise, and comprehensible manner to facilitate understanding and critical evaluation. We present a recommendation for transparent and structured reporting of ML analysis results specifically directed at clinical researchers. Furthermore, we provide a list of key reporting elements with examples that can be used as a template when preparing and submitting ML-based manuscripts for the same audience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Code for the complete model building pipeline should be published in well-documented scripts with computer environment requirements when possible7,8,13,14,16,19,20,24,25,27,33,50,62,64,65, including statements about any restrictions to access or re-use.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
In the citing paper
“and is best informed - a priori - by medical expert knowledge or existing literature13,17,76.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
In the citing paper
“It has been widely recommended to report any measures taken to prevent overfitting3,7,8,13,15.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
In the citing paper
“One commonly referred strategy is feature selection13,15,27,47,76, for which it is explicitly recommended that selection should work independently of model training (unlike in methods as forward and backward selection) and is best informed - a priori - by medical expert knowledge or existing literature13,17,76.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
In the citing paper
“for which it is explicitly recommended that selection should work independently of model training (unlike in methods as forward and backward selection)”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
In the citing paper
“To facilitate transparency and reproducibility of the developmental process, the used computational architecture, high-performance techniques, software packages, and versioning (data, model, configurations and training scripts) should be reported13,14,16,50,64–66.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,319
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.014 · Cited by 2,969
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710
Phenomapping for Novel Classification of Heart Failure With Preserved Ejection Fraction
Shah, S. J., Katz, D. H., Selvaraj, S., Burke, M. A., Yancy, C. W., Gheorghiade, M., Bonow, R. O., Huang, C.-C., & Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.114.010637 · Cited by 1,157
Self-citationReporting of artificial intelligence prediction models
Collins, G. S., & Moons, K. G. M. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30037-6 · Cited by 701
Contemporary Mortality Risk Prediction for Percutaneous Coronary Intervention
Peterson, E. D., Dai, D., DeLong, E. R., Brennan, J. M., Singh, M., Rao, S. V., Shaw, R. E., Roe, M. T., Ho, K. K. L., Klein, L. W., et al. (2010) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2010.02.005 · Cited by 455
Narula, S., Shameer, K., Omar, A. M. S., Dudley, J. T., & Sengupta, P. P. (2016) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2016.08.062 · Cited by 414
Analysis of machine learning techniques for heart failure readmissions
Mortazavi, B. J., Downing, N. S., Bucholz, E. M., Dharmarajan, K., Manhapra, A., Li, S.-X., Negahban, S. N., & Krumholz, H. M. (2016) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.116.003039 · Cited by 401
Self-citationHospital Quality for Acute Myocardial Infarction
Bradley, E. H., Herrin, J. S., Elbel, B. D., McNamara, R. L., Magid, D. J., Nallamothu, B. K., Wang, Y., Normand, S. T., Spertus, J. A., & Krumholz, H. M. (2006) · JAMA · doi:10.1001/jama.296.1.72 · Cited by 370
Congestive heart failure detection using random forest classifier
Mašetić, Z., & Subaşı, A. (2016) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2016.03.020 · Cited by 363
Deep Learning for Prediction of Obstructive Disease From Fast Myocardial Perfusion SPECT
Betancur, J., Commandeur, F., Motlagh, M. E., Sharir, T., Einstein, A. J., Bokhari, S., Fish, M. B., Ruddy, T. D., Kaufmann, P. A., Sinusas, A. J., et al. (2018) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2018.01.020 · Cited by 363
Austin, P. C., Tu, J. V., Ho, J. E., Levy, D. A., & Lee, D. S. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.11.008 · Cited by 340
Kao, D. P., Lewsey, J., Anand, I. S., Massie, B. M., Zile, M. R., Carson, P. E., McKelvie, R. S., Michel Komajda, N., McMurray, J. J. V., & Lindenfeld, J. M. (2015) · European Journal of Heart Failure · doi:10.1002/ejhf.327 · Cited by 276
Self-citationClinical Implications of Chronic Heart Failure Phenotypes Defined by Cluster Analysis
Ahmad, T., Pencina, M. J., Schulte, P. J., O'Brien, E. C., Whellan, D. J., Piña, I. L., Kitzman, D. W., Lee, K. L., O’Connor, C. M., & Felker, G. M. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2014.07.979 · Cited by 262
Characterization and evaluation of similarity measures for pairs of clusterings
Pfitzner, D., Leibbrandt, R. E., & Powers, D. M. W. (2008) · Knowledge and Information Systems · doi:10.1007/s10115-008-0150-6 · Cited by 245
Limestone: High-throughput candidate phenotype generation via tensor factorization
Ho, J. C., Ghosh, J., Steinhubl, S. R., Stewart, W. F., Denny, J. C., Malin, B., & Sun, J. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.07.001 · Cited by 148
Atrial Fibrillation Detection via Accelerometer and Gyroscope of a Smartphone
Lahdenoja, O., Pänkäälä, M., Koivisto, T., Hurnanen, T., Iftikhar, Z., Nieminen, S., Knuutila, T., Saraste, A., Kiviniemi, T., Vasankari, T., et al. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2688473 · Cited by 146
Huang, C.-X., Murugiah, K., Mahajan, S., Li, S.-X., Dhruva, S. S., Haimovich, J., Wang, Y.-F., Schulz, W. L., Testani, J., Wilson, F., et al. (2018) · PLoS Medicine · doi:10.1371/journal.pmed.1002703 · Cited by 115
Self-citationCheng, C.-A., & Chiu, H.-W. (2017) · 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2017.8037381 · Cited by 73
Sani, Z. A., Alizadehsani, R., Habibi, J., Mashayekhi, H., Boghrati, R., Ghandeharioun, A., Khozeimeh, F., & Alizadeh-Sani, F. (2013) · Research in Cardiovascular Medicine · doi:10.5812/cardiovascmed.10888 · Cited by 70
Gonzalez-Beltran, A. N., Li, P., Zhao, J., Ávila-García, M. S., Roos, M., Thompson, M., van der Horst, E., Kaliyaperumal, R., Luo, R., Lee, T., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127612 · Cited by 45
American Heart Association Precision Medicine Platform
Kass‐Hout, T., Stevens, L. M., & Hall, J. L. (2018) · Circulation · doi:10.1161/circulationaha.117.032041 · Cited by 44
Self-citationPersonalizing the Intensity of Blood Pressure Control
Patel, K., Arnold, S. V., Chan, P. S., Tang, Y., Pokharel, Y., Jones, P. G., & Spertus, J. A. (2017) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.117.003624 · Cited by 43
The Concept of Risk in Comparative Effectiveness Research
Fernandes, N., Vist, G., & Bryant, D. (2015) · New England Journal of Medicine · doi:10.1056/nejmc1415933 · Cited by 28
Machine Learning for Electrocardiographic Diagnosis of Left Ventricular Early Diastolic Dysfunction
Bax, J. J., van der Bijl, P., & Delgado, V. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.02.041 · Cited by 16
Treatment Response Classification in Randomized Clinical Trials: A Decision Tree Approach
Venkatesan, P. V. P. (2013) · Indian Journal of Science and Technology · doi:10.17485/ijst/2013/v6i1.3 · Cited by 7
Henderson, J., Ke, J., Ho, J. C., Ghosh, J., & Wallace, B. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9610 · Cited by 6
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
Machine Learning for Healthcare Wearable Devices: The Big Picture
Sabry, F., Eltaras, T., Labda, W., Alzoubi, K., & Malluhi, Q. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/4653923 · Cited by 321
Avoiding common machine learning pitfalls
Lones, M. A. (2024) · Patterns · doi:10.1016/j.patter.2024.101046 · Cited by 298
Metadata
- Authors
- Stevens, L. M., Mortazavi, B. J., Deo, R. C., Curtis, L., & Kao, D. P.
- Affiliations
- American Heart Association (US)
- Texas A&M University (US)
- Brigham and Women's Hospital (US)
- Duke University (US)
- University of Colorado Anschutz (US)
- Venue
- Circulation: Cardiovascular Quality and Outcomes
- Published
- 2020-10
- Volume
- 13
- Issue
- 10
- Pages
- e006556-e006556
- ISSN
- 1941-7705 (ISSN-L)
- 1941-7713
- PMC
- PMC8320533
- OpenAlex
- W3094171951
- Cited by
- 268
- References
- 34
- Citations by year
- 2026: 39 · 2025: 43 · 2024: 72 · 2023: 46 · 2022: 43 · 2021: 24 · 2020: 1
- FWCI
- 11.20
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Circulation Cardiovascular Quality and Outcomes (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Heart, Lung, and Blood Institute (K08HL125725, L30HL110124)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- machine learning, reporting guidelines, clinical research, reproducibility, transparency, peer review, manuscript template, model validation, cardiovascular research
- MeSH terms
- Data Accuracy, Machine Learning, Data Interpretation, Statistical, Humans, Periodicals as Topic, Research Design, Editorial Policies, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
