Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index; 108 of the 519 sources its claims cite were checked for support.

Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review

Open access

Abstract

While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1–3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.

Overview

CoChat Summary

TL;DR

This scoping review identified 72 guidance documents and mapped their recommendations to six phases of the AI-based prediction model lifecycle, from data preparation through implementation. It found substantial guidance for data preparation, development, and validation but less for software development, impact assessment, and implementation, and highlights remaining gaps.

Key findings

  • From 2683 unique hits, 72 relevant guidance documents were included; data extraction yielded 138 keywords mapped to 27 phase-specific topics and 6 phase-overarching topics.
  • Substantial guidance was found for phases 1–3 (data preparation, AIPM development, AIPM validation), while later phases—software development, impact assessment, and implementation—received clearly less attention in the scientific literature.
  • The review provides a six-phase framework for responsible AIPM introduction and identifies gaps including missing a priori sample size guidance for semi-supervised learning, decision trees/random forests, and convolutional neural networks; limited guidance on combining data sources/modalities, non-binary outcomes, AIPM-specific human–AI interaction, workflow integration, cost-effectiveness, accountability, and long-term maintenance/dataset shift.
  • Phase-overarching topics included algorithmic bias, transparency and openness, interpretability, team/stakeholders, security, and risks.

Methods

  • Scoping review with a multi-stage screening strategy; searched PubMed, Web of Science, and ACM Digital Library for literature published January 2000 to January 2021, supplemented by snowballing reference lists and consultation with a convenience sample of 14 AI experts.
  • Inclusion criteria: documents aimed at individuals directly involved in AIPM development, evaluation, and implementation and providing actionable guidance; exclusions included guidance limited to one medical domain or one AI technique, guidance aimed at governing institutions, pre-2000 documents, prerequisites-only guidance, and non-English documents.
  • Two reviewers performed title/abstract and full-text screening, resolving conflicts by consensus; keywords extracted from included documents were mapped to phase-specific and phase-overarching topics and validated with three co-authors.

Limitations

  • Stated: the definitions of “actionable” guidance and the six phases are somewhat arbitrary and mainly informed by experience; individual AIPM applications may deviate from the presented structure.
  • Stated: the databases and sources mostly contain scientific literature and only English documents were included, potentially biasing results toward academic sources and English-speaking countries; this was mitigated by identifying additional gray literature through expert consultation and citation screening.
  • Stated: the consulted expert group was a convenience sample predominantly working in the Netherlands, though it included different occupations, healthcare domains, and academia/industry. Evident: no formal quality appraisal or risk-of-bias assessment of the included guidance documents is reported, and the synthesis is narrative/scoping rather than quantitative.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-021-00549-7

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Flow diagram of screening strategy. This flow diagram displays the screening strategy for the inclusion of guidance documents in this scoping review.
Fig. 1 Flow diagram of screening strategy. This flow diagram displays the screening strategy for the inclusion of guidance documents in this scoping review.CC BY · PubMed Central

Verification

Claims the paper makes

42 Valid · 1 Not valid · 103 Not checked · 57 Own results and interpretations

30 more claims · 3 with their cited sources checked

Create a free account to see all 203 claims and to verify the 411 references still open.

See all claims — free account
  • Probably not a substantial claim. 161
  • It may state two separate claims as one. 50

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 71 references

Research works printed but not linked: 6

  1. Alonso, O. Challenges with label quality for supervised learning. ACM J. Data Inform. Quality 6, 2 (2015).
  2. Eggensperger, K., Lindauer, M. & Hutter, F. Pitfalls and best practices in algorithm configuration. J. Artif. Int. Res. 64, 861–893 (2019).
  3. Kaur, H., Pannu, H. S. & Malhi, A. K. A systematic review on imbalanced data challenges in machine learning: applications and solutions. ACM Comput. Surv. 52, 1–36 (2019).
  4. Institute of Electrical and Electronics Engineers. IEEE approved draft model process for addressing ethical concerns during system design (IEEE 7000-2021). Retrieved from https://standards.ieee.org/standard/7000-2021.html (2021).
  5. International Organization for Standardization. Health informatics—Personal health device communication—Part 10418: Device specialization—International Normalized Ratio (INR) monitor (ISO/IEEE 11073-10418:2014). Retrieved from https://www.iso.org/standard/61897.html (2014).
  6. Food and Drug Administration. Response to NIST workshop and call for position papers on standards and guidelines to enhance software supply chain security. Retrieved from https://www.fda.gov/media/149954/download (2021).

Cited by

See all 570

Metadata

Authors
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., Reitsma, J. B., Steyerberg, E. W., Chavannes, N. H., & Moons, K. G. M.
Affiliations
  • Utrecht University (NL)
  • Leiden University Medical Center (NL)
Corresponding author
Leeuwenberg, Artuur M.
Venue
npj Digital Medicine
Published
2022-01-10
Volume
5
Issue
1
Pages
2-2
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4205805845
Cited by
499
References
71
Citations by year
2026: 93 · 2025: 127 · 2024: 136 · 2023: 105 · 2022: 38
FWCI
6.97
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Leiden Repository (Leiden University) (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Funders
This research was funded by the Ministry of Health, Welfare and Sport.
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
machine learning in healthcare, quality standards, clinical implementation, model validation, data preparation, impact assessment, responsible AI adoption, clinical decision support, software development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. https://app.cochat.ai/papers/10.1038/s41746-021-00549-7
The paper
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., Reitsma, J. B., Steyerberg, E. W., Chavannes, N. H., & Moons, K. G. M. (2022). Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. npj Digital Medicine. https://doi.org/10.1038/s41746-021-00549-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.