The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Evaluation framework to guide implementation of AI systems into healthcare settings
Open accessAbstract
Objectives: To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components. Methods: To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed ‘Translational Evaluation of Healthcare AI (TEHAI)’. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert. Results: TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system. Discussion: One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems. Conclusion: The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“however, its lack of sociotechnical components, such as human-AI integration, culture and organization, and monitoring and support, which are essential for adoption and maximizing utility, is a drawback [3].”
From Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study (2025) · cites this paper as [3]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“however, its lack of sociotechnical components, such as human-AI integration, culture and organization, and monitoring and support, which are essential for adoption and maximizing utility, is a drawback [3].”
From Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Without a shared framework, departments struggle to distinguish safe tools from risky ones,30, 31, 32”
From Artificial Intelligence Readiness in Emergency Medicine: Expert Consensus Opinion for Preparing the Workforce (2026) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Simulation exercises help reveal whether the projected gains are meaningful; if not, ED leaders can move on before investing more time.30,31,54”
From Artificial Intelligence Readiness in Emergency Medicine: Expert Consensus Opinion for Preparing the Workforce (2026) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Emergency physicians who develop a strong understanding of AI principles and gain hands-on experience through workflow simulation case studies can learn and apply clinician-led frameworks for vetting AI technology in the emergency department.30,31,33”
From Artificial Intelligence Readiness in Emergency Medicine: Expert Consensus Opinion for Preparing the Workforce (2026) · cites this paper as [31]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388 · cited 1×
“6–8 13–15 The project team identified key components that could be considered in the framework and developed a draft initial version of the ‘TEHAI’ framework.”
BackgroundCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Recent advances and applications of machine learning in solid-state materials science
Schmidt, J., Marques, M. R. G., Botti, S., & Marques, M. A. L. (2019) · npj Computational Materials · doi:10.1038/s41524-019-0221-0 · Cited by 2,607 · cited 1×
“16 This validation process includes the classical concept of goodness- of-fit, but also”
MethodDeep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421
A governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979
Self-citationCruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Darzi, A., Holmes, C., Yau, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1037-7 · Cited by 933
Artificial intelligence-enabled healthcare delivery
Reddy, S., Fox, J., & Purohit, M. P. (2019) · Journal of the Royal Society of Medicine · doi:10.1177/0141076818815510 · Cited by 635
Self-citationLiu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., Ashrafian, H., Beam, A. L., Chan, A.-W., Collins, G. S., Deeks, A. D. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30218-1 · Cited by 555
Cited by
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“Initial efforts have focused on providing recommendations for the reporting of AI studies for different medical domains or clinical tasks (eg, TRIPOD+AI, 11 CLAIM, 12 CONSORT-AI, 13 DECIDE-AI, 14 PROBAST-AI, 15 CLEAR 16 ).”
Self-citationBias in artificial intelligence algorithms and recommendations for mitigation
Nazer, L. H., Zatarah, R., Waldrip, S., Ke, J. X. C., Moukheiber, M., Khanna, A. K., Hicklen, R. S., Moukheiber, L., Moukheiber, D., Ma, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000278 · Cited by 621
“A number of comprehensive frameworks and checklists have been developed, such as the Translational Evaluation of Healthcare AI (TEHAI) [46], the DECIDE-AI [47], the Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) [48], the prediction model risk of bias assessment tool (PROBAST) [49], an…”
MethodSelf-citationReddy, S. (2024) · Implementation Science · doi:10.1186/s13012-024-01357-9 · Cited by 554
“Col-laborate with regulatory authorities to establish guidelines specific to generative AI in healthcare Establishing procedures for ongoing monitoring and evaluation of the models is crucial in addition to the measures to enable governance for the generative AI models [67, 70].”
MethodSelf-citationLarge language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
Artificial intelligence in nursing and midwifery: A systematic review
O'Connor, S., Yan, Y., Thilo, F. J. S., Felzmann, H., Dowding, D., & Lee, J. J. (2023) · Journal of Clinical Nursing · doi:10.1111/jocn.16478 · Cited by 351
Metadata
- Authors
- Reddy, S., Rogers, W., Makinen, V.-P., Coiera, E., Brown, P., Wenzel, M., Weicken, E., Ansari, S., Mathur, P., Casey, A., & Kelly, B.
- Affiliations
- Deakin University (AU)
- Macquarie University (AU)
- South Australian Health and Medical Research Institute, Adelaide, South Australia, Australia
- Orion Health, Auckland, Auckland, New Zealand
- Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
- Cleveland Clinic (US)
- South Australian Health and Medical Research Institute (AU)
- Corresponding author
- Reddy, Sandeep
- ORCID
- Venue
- BMJ Health & Care Informatics
- Publisher
- BMJ
- Published
- 2021-10
- Volume
- 28
- Issue
- 1
- Pages
- e100444
- ISSN
- 2632-1009 (ISSN-L)
- PMC
- PMC8513218
- OpenAlex
- W3207900783
- Cited by
- 230
- References
- 23
- Citations by year
- 2026: 58 · 2025: 82 · 2024: 56 · 2023: 26 · 2022: 8
- FWCI
- 6.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMJ Health & Care Informatics (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Deakin Research Online (Deakin University) (submittedVersion)
- QUT ePrints (Queensland University of Technology) (submittedVersion)
- Figshare (submittedVersion, cc-by-nc)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- artificial intelligence in healthcare, AI evaluation framework, translational research, artificial intelligence adoption, AI ethics, generalizability, health technology assessment, AI safety
- MeSH terms
- Artificial Intelligence, Delivery of Health Care, Health Facilities, Program Evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation framework to guide implementation of AI systems into healthcare settings. https://app.cochat.ai/papers/10.1136/bmjhci-2021-100444
- The paper
- Reddy, S., Rogers, W., Makinen, V.-P., Coiera, E., Brown, P., Wenzel, M., Weicken, E., Ansari, S., Mathur, P., Casey, A., & Kelly, B. (2021). Evaluation framework to guide implementation of AI systems into healthcare settings. BMJ Health & Care Informatics. https://doi.org/10.1136/bmjhci-2021-100444
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.