CoChat

Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study

Open access

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 19 of the 28 sources its claims cite were checked for support.

Abstract

Background: The integration of artificial intelligence (AI) in health care settings demands a nuanced approach that considers both technical performance and sociotechnical factors. Objective: This study aimed to develop a checklist that addresses the sociotechnical aspects of AI deployment in health care and provides a structured, holistic guide for teams involved in the life cycle of AI systems. Methods: A literature synthesis identified 20 relevant studies, forming the foundation for the Clinical AI Sociotechnical Framework checklist. A modified Delphi study was then conducted with 35 global health care professionals. Participants assessed the checklist’s relevance across 4 stages: “Planning,” “Design,” “Development,” and “Proposed Implementation.” A consensus threshold of 80% was established for each item. IQRs and Cronbach α were calculated to assess agreement and reliability. Results: The initial checklist had 45 questions. Following participant feedback, the checklist was refined to 34 items, and a final round saw 100% consensus on all items (mean score >0.8, IQR 0). Based on the outcome of the Delphi study, a final checklist was outlined, with 1 more question added to make 35 questions in total. Conclusions: The Clinical AI Sociotechnical Framework checklist provides a comprehensive, structured approach to developing and implementing AI in clinical settings, addressing technical and social factors critical for adoption and success. This checklist is a practical tool that aligns AI development with real-world clinical needs, aiming to enhance patient outcomes and integrate smoothly into health care workflows.

Overview

CoChat Summary

TL;DR

This instrument development study synthesized 20 studies and conducted a modified Delphi with 35 global health care professionals to create the Clinical AI Sociotechnical Framework (CASoF) checklist for developing and implementing AI in clinical settings. The final checklist contains 35 questions across planning, design and development, and proposed implementation, with 100% consensus on all items in the final round.

Key findings

  • The literature synthesis identified 20 relevant studies; 14 (65%) addressed specific areas of the AI development cycle, and these were synthesized with a prior primary study into the CASoF domains.
  • The initial checklist had 45 questions; after round 1 feedback it was reduced to 34 questions by merging stages/domains, and 1 more question was added to produce a final 35-question checklist.
  • In the first Delphi round, the mean relevance score exceeded 0.8 on all but one question, and IQR was 0 for all except 3 questions; in the second round, mean score exceeded 0.8 on all questions and IQR was 0 for all questions.
  • The final round achieved 100% consensus on all items (mean score >0.8, IQR 0), and the checklist is organized into 3 stages—Planning; Design and Development; Proposed Implementation—with questions answered “Yes,” “No,” or “Partially Done.”

Methods

  • Literature synthesis: MEDLINE via OVID and Embase were searched June 25-30, 2023, using keywords on AI, frameworks, implementation, design, evaluation, and clinical settings; 573 studies were identified, 19 underwent full-text review, 3 were excluded, 4 were added through citation tracking, yielding 20 studies.
  • Modified Delphi study: conducted January 23-March 14, 2024, with 35 global health care professionals who had advanced degrees and at least 2 years of relevant experience; a pretest involved 5 professionals; panelists remained anonymous to each other and data were collected via Google Forms.
  • Analysis: preliminary 45 questions were rated on a 1-5 Likert scale; consensus was set a priori as mean score >0.8 and IQR 0; descriptive statistics, IQR, Cronbach α, and inductive content analysis were used, with quantitative analysis in Python via JupyterLab.

Limitations

  • Stated: real-world application may be more difficult and less straightforward than the research suggests; the authors call for continuous refinement, broader stakeholder engagement, and future inclusion of more perspectives from underrepresented regions and specialties.
  • Stated: the framework has limitations in capturing the full spectrum of technical challenges, needs, and implications across diverse global health care contexts, and its application may benefit from synergistic use with other existing frameworks.
  • Evident from the text: panelists described the checklist as long and burdensome and suggested cutting the questions in half; the authors intentionally kept it high level, which may limit actionable specificity. The second-round completion count is not reported, which limits interpretation of the final consensus.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/65565

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. The process of developing the checklist.
Figure 1. The process of developing the checklist.CC BY · PubMed Central
Figure 2. PRISMA (Preferred Reporting Items for Systematic Reviews for Meta-Analyses) flowchart.
Figure 2. PRISMA (Preferred Reporting Items for Systematic Reviews for Meta-Analyses) flowchart.CC BY · PubMed Central
Figure 3. The Clinical AI Sociotechnical Framework. AI: artificial intelligence.
Figure 3. The Clinical AI Sociotechnical Framework. AI: artificial intelligence.CC BY · PubMed Central

Verification

Claims the paper makes

13 Valid · 4 Not valid · 9 Not checked · 60 Own results and interpretations

13 more claims

Create a free account to see all 86 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 56
  • It may state two separate claims as one. 20

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

3 of 3

Citations

Cites

Show all 38 references

Metadata

Authors
Owoyemi, A., Osuchukwu, J., Salwei, M. E., & Boyd, A.
Affiliations
  • University of Illinois Chicago (US)
  • University of Cincinnati (US)
  • Vanderbilt University Medical Center (US)
Corresponding author
Owoyemi, Ayomide
Venue
JMIRx Med
Published
2025-02-20
Volume
6
Pages
e65565-e65565
ISSN
2563-6316 (ISSN-L)
OpenAlex
W4407767410
Cited by
18
References
36
Citations by year
2026: 8 · 2025: 9
FWCI
3.48
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIRx Med (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Agency for Healthcare Research and Quality (K01 HS029042)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
artificial intelligence in healthcare, AI implementation, socio-technical factors, checklist development, AI lifecycle, clinical implementation of AI, healthcare professionals' perspectives, Delphi consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.