Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications: A Position Paper from the IMIA Technology Assessment & Quality Development in Health Informatics Working Group and the EFMI Working Group for Assessment of Health Information Systems

Open access

Abstract

Objectives: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. Method: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. Results: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. Conclusion: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1055/s-0039-1677903

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 53 references

Research works printed but not linked: 2

  1. [30]Feigenbaum E Aeditor Autoknowledge: from file server to knowledge serversMEDINFO; 1986; Amsterdam: Elsevier Science Publishers BV 1986
  2. [31]Nykänen P Zvarova J Big data challenges for personalised medicine. Editorial International Journal of Biomedicine and Healthcare 2015 3 01 1

Cited by

See all 347

Metadata

Authors
Magrabi, F., Ammenwerth, E., McNair, J. B., De Keizer, N. F., Hyppönen, H., Nykänen, P., Rigby, M., Scott, P. J., Vehko, T., Wong, Z. S.-Y., & Georgiou, A.
Affiliations
  • Macquarie University (AU)
  • UMIT TIROL - Private University for Health Sciences and Health Technology (AT)
  • Aalborg University (DK)
  • Amsterdam UMC, University of Amsterdam, Department of Medical Informatics, Amsterdam Public Health research institute, The Netherlands
  • Finnish Institute for Health and Welfare (FI)
  • Tampere University (FI)
  • Keele University (GB)
  • University of Portsmouth (GB)
  • St. Luke's International University (JP)
Corresponding author
Magrabi, Farah
Venue
Yearbook of Medical Informatics
Published
2019-04-25
Volume
28
Issue
01
Pages
128-134
ISSN
  • 0943-4747 (ISSN-L)
  • 2364-0502
OpenAlex
W2940562610
Cited by
347
References
53
Citations by year
2026: 57 · 2025: 80 · 2024: 73 · 2023: 48 · 2022: 33 · 2021: 36 · 2020: 17 · 2019: 2
FWCI
1.05
Type
journal article
Language
en
Open access
Locations
  • Yearbook of Medical Informatics (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion, cc-by)
  • Keele Research Repository (Keele University) (submittedVersion, cc-by)
  • VBN Forskningsportal (Aalborg Universitet) (submittedVersion, cc-by-nc-nd)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Tampere University Institutional Repository (Tampere University) (submittedVersion, cc-by-nc-nd)
Funders
Japan Society for the Promotion of Science (18H03336)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
artificial intelligence, clinical decision support, AI evaluation, AI safety, AI monitoring, socio-technical systems, evaluation framework
MeSH terms
Machine Learning, Artificial Intelligence, Evaluation Studies as Topic, Program Evaluation, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications: A Position Paper from the IMIA Technology Assessment & Quality Development in Health Informatics Working Group and the EFMI Working Group for Assessment of Health Information Systems. https://app.cochat.ai/papers/10.1055/s-0039-1677903
The paper
Magrabi, F., Ammenwerth, E., McNair, J. B., De Keizer, N. F., Hyppönen, H., Nykänen, P., Rigby, M., Scott, P. J., Vehko, T., Wong, Z. S.-Y., & Georgiou, A. (2019). Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications: A Position Paper from the IMIA Technology Assessment & Quality Development in Health Informatics Working Group and the EFMI Working Group for Assessment of Health Information Systems. Yearbook of Medical Informatics. https://doi.org/10.1055/s-0039-1677903

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.