Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review
Open accessThe DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index; 108 of the 519 sources its claims cite were checked for support.
Abstract
While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1–3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.
Overview
CoChat Summary
TL;DR
This scoping review identified 72 guidance documents and mapped their recommendations to six phases of the AI-based prediction model lifecycle, from data preparation through implementation. It found substantial guidance for data preparation, development, and validation but less for software development, impact assessment, and implementation, and highlights remaining gaps.
Key findings
- From 2683 unique hits, 72 relevant guidance documents were included; data extraction yielded 138 keywords mapped to 27 phase-specific topics and 6 phase-overarching topics.
- Substantial guidance was found for phases 1–3 (data preparation, AIPM development, AIPM validation), while later phases—software development, impact assessment, and implementation—received clearly less attention in the scientific literature.
- The review provides a six-phase framework for responsible AIPM introduction and identifies gaps including missing a priori sample size guidance for semi-supervised learning, decision trees/random forests, and convolutional neural networks; limited guidance on combining data sources/modalities, non-binary outcomes, AIPM-specific human–AI interaction, workflow integration, cost-effectiveness, accountability, and long-term maintenance/dataset shift.
- Phase-overarching topics included algorithmic bias, transparency and openness, interpretability, team/stakeholders, security, and risks.
Methods
- Scoping review with a multi-stage screening strategy; searched PubMed, Web of Science, and ACM Digital Library for literature published January 2000 to January 2021, supplemented by snowballing reference lists and consultation with a convenience sample of 14 AI experts.
- Inclusion criteria: documents aimed at individuals directly involved in AIPM development, evaluation, and implementation and providing actionable guidance; exclusions included guidance limited to one medical domain or one AI technique, guidance aimed at governing institutions, pre-2000 documents, prerequisites-only guidance, and non-English documents.
- Two reviewers performed title/abstract and full-text screening, resolving conflicts by consensus; keywords extracted from included documents were mapped to phase-specific and phase-overarching topics and validated with three co-authors.
Limitations
- Stated: the definitions of “actionable” guidance and the six phases are somewhat arbitrary and mainly informed by experience; individual AIPM applications may deviate from the presented structure.
- Stated: the databases and sources mostly contain scientific literature and only English documents were included, potentially biasing results toward academic sources and English-speaking countries; this was mitigated by identifying additional gray literature through expert consultation and citation screening.
- Stated: the consulted expert group was a convenience sample predominantly working in the Netherlands, though it included different occupations, healthcare domains, and academia/industry. Evident: no formal quality appraisal or risk-of-bias assessment of the included guidance documents is reported, and the synthesis is narrative/scoping rather than quantitative.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
42 Valid · 1 Not valid · 103 Not checked · 57 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“These phases are based on the predominant phases in clinical prediction model research4,5.”
§ Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review > Introduction
What the sources say
- Supportedcites [4] Steyerberg EW Prognosis Research Strategy (PROGRESS) 3: prognostic model research PLoS Med. 2013 10 e1001381 23393430 PMC3564751 source
The Summary Points of PROGRESS 3 state that 'Prognostic model research has three main phases: model development (including internal validation), external validation, and investigations of impact in clinical practice', and that most publications describe model development with few external validation and very few impact studies; the Conclusion repeats the three-phase framing. The paper also explicitly equates prognostic models with 'risk (or clinical) prediction model' in its Introduction, so it does treat these as the predominant phases of clinical prediction model research that the citing authors' AIPM phases are said to build on. The number of citing phases (six) is the citing authors' own construction, not something attributed to this source, so no mismatch arises. (LLM, conf=0.85)
- Not checked yetcites [5] Moons KG Altman DG Vergouwe Y Royston P Prognosis and prognostic research: application and impact of prognostic models in clinical practice BMJ 2009 338 b606 19502216 source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It should be noted that when the sample size is simply too small, even penalization methods have been shown ineffective to mitigate overfitting83,84.”
§ Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review > Results > Phase 2. Development of the AIPM
What the sources say
- Partly supportedcites [83] Van Calster B van Smeden M De Cock B Steyerberg EW Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study Stat. Methods Med. Res. 2020 29 3166 3178 32401702 sourceExperimental — accuracy not yet measured
In the section labeled 3.4 (page not available), the paper says shrinkage is particularly variable at low sample sizes and does not justify developing a model with a smaller sample. But it also says shrinkage works on average and Firth’s penalized likelihood almost surely improved performance over maximum likelihood. The claim overstates this as penalization methods being ineffective to mitigate overfitting. (LLM, conf=0.84)
- Partly supportedcites [84] Riley RD Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small J. Clin. Epidemiol. 2021 132 88 96 33307188 PMC8026952 sourceExperimental — accuracy not yet measured
The Summary says penalization and shrinkage are not a solution to small effective sample sizes, and the Discussion says they may be unreliable when overfitting is greatest. But the Introduction also says penalization may be effective on average; the paper does not establish that it is categorically ineffective at mitigating overfitting in small samples. (LLM, conf=0.86)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unlike discrimination and calibration, DCA derives the clinical utility from the predictive performance5,6,17,49,68,72.”
§ Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review > Results > Phase 2. Development of the AIPM
What the sources say
- Not checked yetcites [5] Moons KG Altman DG Vergouwe Y Royston P Prognosis and prognostic research: application and impact of prognostic models in clinical practice BMJ 2009 338 b606 19502216 source
- Partly supportedcites [6] Luo W Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view J. Med. Internet Res. 2016 18 e323 27986644 PMC5238707 sourceExperimental — accuracy not yet measured
The Results section (Table 5, item 10; page unavailable) recommends reporting clinical implications derived from predictive performance. It does not identify decision curve analysis (DCA) as the method for doing so or contrast DCA with discrimination and calibration. (LLM, conf=0.78)
- Supportedcites [17] Steyerberg EW Vergouwe Y Towards better clinical prediction models: seven steps for development and an ABCD for validation Eur. Heart J. 2014 35 1925 1931 24898551 PMC4155437 source
The Introduction’s “Decision-curve analysis” section says calibration and discrimination do not assess clinical usefulness, whereas decision thresholds and net benefit can be used to assess whether a model leads to better decisions. Table 2 identifies decision-curve analysis as the clinical-usefulness measure for model performance. (LLM, conf=0.91)
- Not checked yetcites [49] Steyerberg, E. W. Clinical Prediction Models (Springer Nature, 2009). source
- Supportedcites [68] Kappen TH Evaluating the impact of prediction models: lessons learned, challenges, and recommendations Diagn. Progn. Res. 2018 2 11 31093561 PMC6460651 source
The Background says discrimination and calibration can indicate predictive performance but do not guarantee improved clinical decisions or outcomes. In “Design of the impact study” (page unavailable), the paper explains that decision-analytic techniques estimate a model’s potential impact using predictive accuracy together with decision thresholds and other assumptions; its cited examples include decision curve analysis. This supports the distinction, provided “clinical utility” means estimated rather than demonstrated benefit. (LLM, conf=0.83)
- Supportedcites [72] Kelly CJ Karthikesalingam A Suleyman M Corrado G King D Key challenges for delivering clinical impact with artificial intelligence BMC Med. 2019 17 195 31665002 PMC6821018 source
In “Metrics often do not reflect clinical applicability” (page unavailable), the paper contrasts conventional predictive-performance measures with whether a model benefits patient care. It identifies decision curve analysis as an approach that aims to quantify the net benefit of using a model to guide actions, supporting the cited distinction. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“One commonly referred strategy is feature selection13,15,27,47,76, for which it is explicitly recommended that selection should work independently of model training (unlike in methods as forward and backward selection) and is best informed - a priori - by medical expert knowledge or existing literature13,17,76.”
§ Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review > Results > Phase 2. Development of the AIPM
What the sources say
- Supportedcites [13] Stevens LM Mortazavi BJ Deo RC Curtis L Kao DP Recommendations for reporting machine learning analyses in clinical research Circ. Cardiovasc Qual. Outcomes 2020 13 e006556 33079589 PMC8320533 source
The Feature Engineering, Selection, and Transformation section (page not provided) says that excessive variables increase spurious findings, models perform best with less redundancy, and feature selection can eliminate that redundancy. This supports the paraphrase that feature selection is a strategy for reducing overfitting risk. (LLM, conf=0.82)
- Not checked yetcites [15] Food and Drug Administration. Computer-assisted detection devices applied to radiology images and radiology device data—premarket notification [510(k)] submissions. Retrieved from https://www.fda.gov/regulatory-information/search-fda-guidance-documents/computer-assisted-detection-devices-applied-radiology-images-and-radiology-device-data-premarket (2012).
- Partly supportedcites [27] Cearns M Hahn T Baune BT Recommendations and future directions for supervised machine learning in psychiatry Transl. Psychiatry 2019 9 271 31641106 PMC6805872 sourceExperimental — accuracy not yet measured
The Model training and evaluation section recommends careful feature selection to avoid data leakage. The Future goals section explains that high-dimensional data and too many model parameters can lead to overfitting, but our copy does not explicitly describe feature selection as a commonly used strategy to prevent overfitting. (LLM, conf=0.78)
- Not checked yetcites [47] Bhaskar H Hoyle DC Singh S Machine learning in bioinformatics: a brief survey and recommendations for practitioners Comput Biol. Med. 2006 36 1104 1125 16226240 source
- Not checked yetcites [76] Aliferis CF Statnikov A Tsamardinos I Challenges in the analysis of mass-throughput data: a technical commentary from the statistical machine learning perspective Cancer Inf. 2007 2 133 162 PMC2675497 19458765
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Similar to internal validation of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, specificity, positive and negative predictive values), calibration (calibration plot)5,6,13,17,49,56,78, and Decision Curve Analysis5,6,17,49,68,72.”
§ Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review > Results > Phase 3. Validation of the AIPM
What the sources say
- Not checked yetcites [5] Moons KG Altman DG Vergouwe Y Royston P Prognosis and prognostic research: application and impact of prognostic models in clinical practice BMJ 2009 338 b606 19502216 source
- Partly supportedcites [6] Luo W Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view J. Med. Internet Res. 2016 18 e323 27986644 PMC5238707 sourceExperimental — accuracy not yet measured
The Results checklist recommends external validation when possible and lists sensitivity, specificity, predictive values, area under the ROC curve, and a calibration plot as classification validation metrics. It does not identify Decision Curve Analysis as a validation method, so that part of the claim is unsupported by this source. Page numbers are unavailable in our copy. (LLM, conf=0.87)
- Partly supportedcites [13] Stevens LM Mortazavi BJ Deo RC Curtis L Kao DP Recommendations for reporting machine learning analyses in clinical research Circ. Cardiovasc Qual. Outcomes 2020 13 e006556 33079589 PMC8320533 sourceExperimental — accuracy not yet measured
The Study Design section (Determining Measures for Evaluation; page not provided) lists ROC AUC, sensitivity and positive predictive value as supervised-model metrics and discusses calibration in external validation. The External Model Evaluation section also recommends reporting calibration where applicable. Neither passage identifies Decision Curve Analysis or a calibration plot as an external-validation method. (LLM, conf=0.86)
- Supportedcites [17] Steyerberg EW Vergouwe Y Towards better clinical prediction models: seven steps for development and an ABCD for validation Eur. Heart J. 2014 35 1925 1931 24898551 PMC4155437 source
The Introduction’s “An ABCD for model validation” section proposes measures of calibration, discrimination, and clinical usefulness and illustrates them using external validation. Table 2 identifies a calibration plot, ROC-based discrimination, and decision-curve analysis as corresponding measures. (LLM, conf=0.94)
- Not checked yetcites [49] Steyerberg, E. W. Clinical Prediction Models (Springer Nature, 2009). source
- Not checked yetcites [56] Park SH Han K Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction Radiology 2018 286 800 809 29309734 source
- Not checked yetcites [78] Altman DG Vergouwe Y Royston P Moons KGM Prognosis and prognostic research: validating a prognostic model BMJ 2009 338 b605 19477892 source
- Partly supportedcites [68] Kappen TH Evaluating the impact of prediction models: lessons learned, challenges, and recommendations Diagn. Progn. Res. 2018 2 11 31093561 PMC6460651 sourceExperimental — accuracy not yet measured
In the Background section, the paper treats discrimination and calibration as measures of predictive performance in a new setting. In “Design of the impact study,” it discusses decision-analytic studies as a way to estimate the potential impact of an already validated model, not as a measure of external validation. The claim therefore folds a separate assessment into external validation. (LLM, conf=0.86)
- Partly supportedcites [72] Kelly CJ Karthikesalingam A Suleyman M Corrado G King D Key challenges for delivering clinical impact with artificial intelligence BMC Med. 2019 17 195 31665002 PMC6821018 sourceExperimental — accuracy not yet measured
The 'Metrics often do not reflect clinical applicability' section discusses ROC area, sensitivity, specificity, predictive values, and decision curve analysis as ways to assess model performance or clinical benefit. The 'Challenges in generalisation to new populations and settings' section calls for external validation. Our copy does not identify calibration or calibration plots as a basis for that validation, so it supports only part of the attributed list. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
30 more claims · 3 with their cited sources checked
Create a free account to see all 203 claims and to verify the 411 references still open.
- Probably not a substantial claim. 161
- It may state two separate claims as one. 50
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the clinical translation of multimodal fusion models requires rigorous validation through real-world trials, covering experimental design, data processing, modeling, testing, and safety assessment (de Hond et al., 2022).”
From Multimodal neuroimaging and AI integration in cognitive disorders: advances, challenges, and future directions for precision medicine (2026) · cites this paper as de Hond et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As AI solutions have emerged, a wide variety of governance best practices in the form of frameworks, governance structures, checklists, and guidelines have been proposed7–23.”
From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most articles briefly addressed the importance of clear objectives for AI implementation in healthcare, with only a few offering actionable guidance9,11,14,17,21.”
From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI tools that learn from their outputs through feedback loops must be closely monitored, as unchecked adjustments can amplify errors or biases over time14,19.”
From Advancing healthcare AI governance through a comprehensive maturity model based on systematic review (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing strategies for testing and evaluating AI models in the context of the workflows they are part of consist primarily of reporting guidelines and checklists that focus on the properties of the model, including performance metrics, how it will be deployed, and potential outcomes 6,7 .”
From Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems (2024) · cites this paper as [7]
Experimental — accuracy not yet measured
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 8×
“…with a possibility to discuss whether the associations on which the AIPM relies are likely to remain stable 8,27,61 , help to identify algorithmic bias 7,22,24,27,40,43 , provide information on where the AIPM could be most easily attacked 27 , or how the AIPM may behave under dataset shift 7 .”
BackgroundMethodInfluentialThe global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“This sets our review apart from other work that was previously undertaken (e.g., 32,123,124 ).”
ResultExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · cited 3×
“For regression-based methods 48 , and a selection of machine-learning-based methods 47 , technique-speci fi c a priori sample size calculations are available, although for many model architectures and settings (e.g., semi-supervised learning, decision trees, or convolutional neural networks) no…”
BackgroundMethodSelf-citationAntlanger, M., Aschauer, S., Kopecky, C., Hecking, M., Kovarik, J. J., Werzowa, J., Mascherbauer, J., Genser, B., Säemann, M. D., & Bonderman, D. (2017) · Kidney and Blood Pressure Research · doi:10.1159/000473868 · Cited by 2,808
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · cited 3×
“Still, AIPM updating, site-speci fi c training or recalibration might be needed to adapt an existing AIPM to a different healthcare setting 3,5,9,60,68,72 .”
BackgroundMethodSelf-citationGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 3×
“It is recommended that the amount of collected data is suf fi ciently large for the intended purpose 6,8,12,15,20,22,26,46 – 49 , is ideally prespeci fi ed 8 and should be clearly reported 3,14,37,46,50 .”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 8×
“Multiple sources state that the design should match social norms, and make sure its presentation does not reinforce stereotypes (e.g., regarding a prespeci fi ed fairness position or bias impact statement, see phase 2) 22,24,26,32,99 .”
BackgroundInfluentialTowards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Self-citationThe Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911 · cited 1×
“This sets our review apart from other work that was previously undertaken (e.g., 32,123,124 ).”
ResultThe Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907 · cited 1×
“This sets our review apart from other work that was previously undertaken (e.g., 32,123,124 ).”
ResultPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · cited 2×
“These phases are based on the predominant phases in clinical prediction model research 4,5 .”
BackgroundResultSelf-citationSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Self-citationChecklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationDo no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Risk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195
Self-citationLuo, W., Phung, D., Tran, T., Gupta, S., Rana, S., Karmakar, C., Shilton, A., Yearwood, J., Dimitrova, N., Ho, T. B., et al. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5870 · Cited by 988
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934
Self-citationCruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Darzi, A., Holmes, C., Yau, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1037-7 · Cited by 933
(2015) · neural information processing systems · Cited by 912
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894 · cited 9×
“…of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, speci fi city, positive and negative predictive values), calibration (calibration plot) 5,6,13,17,49,56,78 , and Decision Curve Analysis 5,6,17,49,68,72 .”
BackgroundMethodInfluentialMoons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889
Self-citationTowards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Clinical Chemistry · doi:10.1373/49.1.1 · Cited by 862
Self-citationNational electrical manufacturers association
(1926) · Journal of the A.I.E.E. · doi:10.1109/jaiee.1926.6537344 · Cited by 860
Leakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Establishment of Best Practices for Evidence for Prediction: A Review
Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.3671 · Cited by 738
Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020) · BMJ · doi:10.1136/bmj.m3164 · Cited by 702
Calibrating Probability with Undersampling for Unbalanced Classification
Pozzolo, A. D., Caelen, O., Johnson, R. A., & Bontempi, G. (2015) · 2015 IEEE Symposium Series on Computational Intelligence · doi:10.1109/ssci.2015.33 · Cited by 667
Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1041-y · Cited by 665 · cited 8×
“The literature stresses that detailed description of any preprocessing steps applied to the raw data should be provided, including software used to perform the processing steps 3,6,7,9,13 – 15, 22, 50, 61, 62, Data coding standards .”
BackgroundMethodInfluentialVergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. C., & Habbema, J. D. F. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.06.017 · Cited by 636
Self-citationVollmer, S., Mateen, B. A., Bohner, G., Király, F. J., Ghani, R., Jonsson, P., Cumbers, S., Jonas, A., McAllister, K. S. L., Myles, P., et al. (2020) · BMJ · doi:10.1136/bmj.l6927 · Cited by 497
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439
Self-citationMinimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400
Self-citationAI in Health: State of the Art, Challenges, and Future Directions
Wang, F., & Preininger, A. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677908 · Cited by 398
Magrabi, F., Ammenwerth, E., McNair, J. B., De Keizer, N. F., Hyppönen, H., Nykänen, P., Rigby, M., Scott, P. J., Vehko, T., Wong, Z. S.-Y., et al. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677903 · Cited by 347
Ryan, M., & Stahl, B. C. (2021) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-12-2019-0138 · Cited by 310 · cited 10×
“Identifying and involving the right expertise and stakeholders in each consecutive phase of the AIPM development, evaluation and implementation cycle is crucial for its success in daily healthcare practice 8,18,22,23,26,32,58,79,88 .”
BackgroundMethodInfluentialPresenting machine learning model information to clinical end users with model facts labels
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0253-3 · Cited by 279 · cited 1×
“Providing a model facts label should be provided to the end users is advised, including the system ’ s technical speci fi cations, statistical working, limitations, fairness criteria and validation, implementation disclaimer, and links to process logs 22,101 .”
BackgroundEstimating Dataset Size Requirements for Classifying DNA Microarray Data
Mukherjee, S., Tamayo, P., Rogers, S., Rifkin, R., Engle, A., Campbell, C., Golub, T. R., & Mesirov, J. P. (2003) · Journal of Computational Biology · doi:10.1089/106652703321825928 · Cited by 278
Recommendations for Reporting Machine Learning Analyses in Clinical Research
Stevens, L. M., Mortazavi, B. J., Deo, R. C., Curtis, L., & Kao, D. P. (2020) · Circulation: Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.006556 · Cited by 268 · cited 10×
“The literature stresses that detailed description of any preprocessing steps applied to the raw data should be provided, including software used to perform the processing steps 3,6,7,9,13 – 15, 22, 50, 61, 62, Data coding standards .”
BackgroundMethodInfluentialStandard IEC 62304 - Medical device software - Software lifecycle processes
Jordan, P. (2006) · IET Seminar on Software for Medical Devices · doi:10.1049/ic:20060141 · Cited by 244
Clinical prediction models: diagnosis versus prognosis
van Smeden, M., Reitsma, J. B., Riley, R. D., Collins, G. S., & Moons, K. G. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.009 · Cited by 238
Self-citationPredictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223
Self-citationEvaluating the impact of prediction models: lessons learned, challenges, and recommendations
Kappen, T. H., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., Vergouwe, Y., & Moons, K. G. M. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0033-6 · Cited by 215 · cited 10×
“…of the AIPM, external AIPM model validation can be based on discrimination (area under the receiver operating characteristic curve, sensitivity, speci fi city, positive and negative predictive values), calibration (calibration plot) 5,6,13,17,49,56,78 , and Decision Curve Analysis 5,6,17,49,68,72 .”
BackgroundMethodInfluentialSelf-citationThree myths about risk thresholds for prediction models
on behalf of the Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Wynants, L., van Smeden, M., McLernon, D. J., Timmerman, D., Steyerberg, E. W., & Van Calster, B. (2019) · BMC Medicine · doi:10.1186/s12916-019-1425-3 · Cited by 213
Self-citationLarson, D. B., Harvey, H., Rubin, D. L., Irani, N., Tse, J. R., & Langlotz, C. P. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.09.060 · Cited by 198 · cited 10×
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting 3,7,8,10,16,17,27,56,68,72,79 .”
BackgroundMethodInfluentialMinimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Rodríguez-Barroso, N., Stipcich, G., Jiménez-López, D., Ruiz-Millán, J. A., Martínez-Cámara, E., González-Seco, G., Luzón, M. V., Veganzones, M. A., & Herrera, F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.07.009 · Cited by 148
A review of statistical updating methods for clinical prediction models
Su, T.-L., Jaki, T., Hickey, G. L., Buchan, I., & Sperrin, M. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280215626466 · Cited by 147
Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.005 · Cited by 142
Forcier, M. B., Gallois, H., Mullan, S., & Joly, Y. (2019) · Journal of Law and the Biosciences · doi:10.1093/jlb/lsz013 · Cited by 135
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131
Self-citationRecommendations and future directions for supervised machine learning in psychiatry
Cearns, M., Hahn, T., & Baune, B. T. (2019) · Translational Psychiatry · doi:10.1038/s41398-019-0607-2 · Cited by 127
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127 · cited 10×
“…logging and traceability of predictions and decisions is required and the AIPM interface should provide sharing of performance data with end users to enable ongoing monitoring of both individual and aggregated cases, quickly highlighting any signi fi cant deviations in performance 10,26,61,66 .”
BackgroundMethodInfluentialVan Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280220921415 · Cited by 120 · cited 1×
“It should be noted that when the sample size is simply too small, even penalization methods have been shown ineffective to mitigate over fi tting 83,84 .”
Self-citationMcCradden, M. D., Joshi, S., Anderson, J. A., Mazwi, M., Goldenberg, A., & Zlotnik Shaul, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa085 · Cited by 111
Machine learning in bioinformatics: A brief survey and recommendations for practitioners
Bhaskar, H., Hoyle, D. C., & Singh, S. (2006) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2005.09.002 · Cited by 109 · cited 9×
“For regression-based methods 48 , and a selection of machine-learning-based methods 47 , technique-speci fi c a priori sample size calculations are available, although for many model architectures and settings (e.g., semi-supervised learning, decision trees, or convolutional neural networks) no…”
BackgroundMethodInfluentialGuideline for good evaluation practice in health informatics (GEP-HI)
Nykänen, P., Brender, J., Talmon, J., de Keizer, N., Rigby, M., Beuscart-Zephir, M.-C., & Ammenwerth, E. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.08.004 · Cited by 105 · cited 7×
“…into the trial setting is highly recommended, including onsite and offsite requirements, version number and other technical speci fi cations 28,64,65 , but also the human-AI interaction involved (e.g., assistive versus directive, see phase 4) 46,64 and the patient treatment strategy…”
BackgroundResultInfluentialLiaw, S.-T., Liyanage, H., Kuziemsky, C., Terry, A. L., Schreiber, R., Jonnagaddala, J., & de Lusignan, S. (2020) · Yearbook of Medical Informatics · doi:10.1055/s-0040-1701980 · Cited by 76 · cited 9×
“Legislation may require identi fi cation of the right legal basis (such as informed consent) for processing con fi dential information of individuals 12,20,26,32,33,36,37 .”
BackgroundInfluentialNwe, K., Larsen, M. E., Nelissen, N., & Wong, D. (2020) · Journal of Medical Internet Research · doi:10.2196/17457 · Cited by 58
From Code to Bedside: Implementing Artificial Intelligence Using Quality Improvement Methods
Smith, M., Sattler, A., Hong, G., & Lin, S. (2021) · Journal of General Internal Medicine · doi:10.1007/s11606-020-06394-w · Cited by 51 · cited 7×
“This type of study consists of repeated live clinical tests in which variation is key to understanding the functionality of the technology and work fl ow 11,18 .”
BackgroundMethodInfluentialAdaptive sample size determination for the development of clinical prediction models
Christodoulou, E., van Smeden, M., Edlinger, M., Timmerman, D., Wanitschek, M. M., Steyerberg, E. W., & CALSTER, B. V. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-021-00096-5 · Cited by 46
Self-citationResponsible AI by Design in Practice
Benjamins, R., Barbado, A., & Sierra, D. (2019) · arXiv · doi:10.48550/arxiv.1909.12838 · Cited by 38
Aliferis, C., Statnikov, A., & Tsamardinos, I. (2006) · Cancer Informatics · doi:10.1177/117693510600200004 · Cited by 36
Ray, J. M., Ratwani, R. M., Sinsky, C. A., Frankel, R. M., Friedberg, M. W., Powsner, S. M., Rosenthal, D. I., Wachter, R. M., & Melnick, E. R. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz098 · Cited by 34 · cited 10×
“Identifying and involving the right expertise and stakeholders in each consecutive phase of the AIPM development, evaluation and implementation cycle is crucial for its success in daily healthcare practice 8,18,22,23,26,32,58,79,88 .”
BackgroundMethodInfluentialWeikert, T., Francone, M., Abbara, S., Baessler, B., Choi, B. W., Gutberlet, M., Hecht, E. M., Loewe, C., Mousseaux, E., Natale, L., et al. (2021) · European Radiology · doi:10.1007/s00330-020-07417-0 · Cited by 34 · cited 8×
“…recommended that already from the start the AIPM development team must cover a multidisciplinary technical, methodological and medical expertise 8,11,17,20,23,26,32,71 , consider data and project management 8,11,18,20,26 , and attend to the diversity of the anticipated end users of the AIPM…”
BackgroundMethodInfluentialAliferis, C., Statnikov, A., & Tsamardinos, I. (2007) · PubMed · Cited by 28
Beyond transparency: a proposed framework for accountability in decision-making AI systems
Berscheid, J., & Roewer-Despres, F. (2019) · AI Matters · doi:10.1145/3340470.3340476 · Cited by 22
CALSTER, B. V., Steyerberg, E. W., & Collins, P. G. S. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2019.0597 · Cited by 15
Self-citation
Research works printed but not linked: 7
- National Academy of Medicine. Artificial Intelligence In Health Care: The Hope, The Hype, The Promise, The Peril (National Academy of Medicine, 2019). 39146448
- Alonso O Challenges with label quality for supervised learning ACM J. Data Inform. Quality 2015 6 2
- Eggensperger K Lindauer M Hutter F Pitfalls and best practices in algorithm configuration J. Artif. Int. Res. 2019 64 861 893
- Kaur H Pannu HS Malhi AK A systematic review on imbalanced data challenges in machine learning: applications and solutions ACM Comput. Surv. 2019 52 1 36
- Institute of Electrical and Electronics Engineers. IEEE approved draft model process for addressing ethical concerns during system design (IEEE 7000-2021). Retrieved from https://standards.ieee.org/standard/7000-2021.html (2021).
- International Organization for Standardization. Health informatics—Personal health device communication—Part 10418: Device specialization—International Normalized Ratio (INR) monitor (ISO/IEEE 11073-10418:2014). Retrieved from https://www.iso.org/standard/61897.html (2014).
- Food and Drug Administration. Response to NIST workshop and call for position papers on standards and guidelines to enhance software supply chain security. Retrieved from https://www.fda.gov/media/149954/download (2021).
Cited by
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationLekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
SKCV: Stratified K-fold cross-validation on ML classifiers for predicting cervical cancer
Prusty, S., Patnaik, S., & Dash, S. K. (2022) · Frontiers in Nanotechnology · doi:10.3389/fnano.2022.972421 · Cited by 248
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246
Self-citation
Metadata
- Authors
- de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., Reitsma, J. B., Steyerberg, E. W., Chavannes, N. H., & Moons, K. G. M.
- Affiliations
- Utrecht University (NL)
- Leiden University Medical Center (NL)
- Corresponding author
- Leeuwenberg, Artuur M.
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2022-01-10
- Volume
- 5
- Issue
- 1
- Pages
- 2-2
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC8748878
- OpenAlex
- W4205805845
- Cited by
- 499
- References
- 71
- Citations by year
- 2026: 93 · 2025: 127 · 2024: 136 · 2023: 105 · 2022: 38
- FWCI
- 6.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- This research was funded by the Ministry of Health, Welfare and Sport.
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- machine learning in healthcare, quality standards, clinical implementation, model validation, data preparation, impact assessment, responsible AI adoption, clinical decision support, software development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
