Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 19 of the 28 sources its claims cite were checked for support.
Abstract
Background: The integration of artificial intelligence (AI) in health care settings demands a nuanced approach that considers both technical performance and sociotechnical factors. Objective: This study aimed to develop a checklist that addresses the sociotechnical aspects of AI deployment in health care and provides a structured, holistic guide for teams involved in the life cycle of AI systems. Methods: A literature synthesis identified 20 relevant studies, forming the foundation for the Clinical AI Sociotechnical Framework checklist. A modified Delphi study was then conducted with 35 global health care professionals. Participants assessed the checklist’s relevance across 4 stages: “Planning,” “Design,” “Development,” and “Proposed Implementation.” A consensus threshold of 80% was established for each item. IQRs and Cronbach α were calculated to assess agreement and reliability. Results: The initial checklist had 45 questions. Following participant feedback, the checklist was refined to 34 items, and a final round saw 100% consensus on all items (mean score >0.8, IQR 0). Based on the outcome of the Delphi study, a final checklist was outlined, with 1 more question added to make 35 questions in total. Conclusions: The Clinical AI Sociotechnical Framework checklist provides a comprehensive, structured approach to developing and implementing AI in clinical settings, addressing technical and social factors critical for adoption and success. This checklist is a practical tool that aligns AI development with real-world clinical needs, aiming to enhance patient outcomes and integrate smoothly into health care workflows.
Overview
CoChat Summary
TL;DR
This instrument development study synthesized 20 studies and conducted a modified Delphi with 35 global health care professionals to create the Clinical AI Sociotechnical Framework (CASoF) checklist for developing and implementing AI in clinical settings. The final checklist contains 35 questions across planning, design and development, and proposed implementation, with 100% consensus on all items in the final round.
Key findings
- The literature synthesis identified 20 relevant studies; 14 (65%) addressed specific areas of the AI development cycle, and these were synthesized with a prior primary study into the CASoF domains.
- The initial checklist had 45 questions; after round 1 feedback it was reduced to 34 questions by merging stages/domains, and 1 more question was added to produce a final 35-question checklist.
- In the first Delphi round, the mean relevance score exceeded 0.8 on all but one question, and IQR was 0 for all except 3 questions; in the second round, mean score exceeded 0.8 on all questions and IQR was 0 for all questions.
- The final round achieved 100% consensus on all items (mean score >0.8, IQR 0), and the checklist is organized into 3 stages—Planning; Design and Development; Proposed Implementation—with questions answered “Yes,” “No,” or “Partially Done.”
Methods
- Literature synthesis: MEDLINE via OVID and Embase were searched June 25-30, 2023, using keywords on AI, frameworks, implementation, design, evaluation, and clinical settings; 573 studies were identified, 19 underwent full-text review, 3 were excluded, 4 were added through citation tracking, yielding 20 studies.
- Modified Delphi study: conducted January 23-March 14, 2024, with 35 global health care professionals who had advanced degrees and at least 2 years of relevant experience; a pretest involved 5 professionals; panelists remained anonymous to each other and data were collected via Google Forms.
- Analysis: preliminary 45 questions were rated on a 1-5 Likert scale; consensus was set a priori as mean score >0.8 and IQR 0; descriptive statistics, IQR, Cronbach α, and inductive content analysis were used, with quantitative analysis in Python via JupyterLab.
Limitations
- Stated: real-world application may be more difficult and less straightforward than the research suggests; the authors call for continuous refinement, broader stakeholder engagement, and future inclusion of more perspectives from underrepresented regions and specialties.
- Stated: the framework has limitations in capturing the full spectrum of technical challenges, needs, and implications across diverse global health care contexts, and its application may benefit from synergistic use with other existing frameworks.
- Evident from the text: panelists described the checklist as long and burdensome and suggested cutting the questions in half; the authors intentionally kept it high level, which may limit actionable specificity. The second-round completion count is not reported, which limits interpretation of the final consensus.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
13 Valid · 4 Not valid · 9 Not checked · 60 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“however, its lack of sociotechnical components, such as human-AI integration, culture and organization, and monitoring and support, which are essential for adoption and maximizing utility, is a drawback [3].”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Discussion > Principal Findings
What the sources say
- Disputedcites [3] Reddy, Evaluation framework to guide implementation of AI systems into healthcare settings, BMJ Health Care Inform sourceExperimental — accuracy not yet measured
The Results > Adoption section explicitly includes integration with clinical workflows and considers acceptance by clinicians and patients. Results > Utility includes ongoing monitoring mechanisms, and Results > Adoption recommends checks during an ongoing monitoring phase. The copy does not establish that TEHAI covers organizational culture or support, but it contradicts the claim that the checklist lacks the listed components as a whole. Page numbers are unavailable in our copy. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“Salwei and Carayon [1] developed a sociotechnical systems framework for AI that acknowledges the social and technical aspects of work that relate to the successful design and implementation of AI.”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Introduction
What the sources say
- Supportedcites [1] Salwei, A sociotechnical systems framework for the application of artificial intelligence in health care delivery, J Cogn Eng Decis Mak source
The Conclusion says the authors use a work system model to outline sociotechnical considerations for designing AI and to help designers and implementers consider how AI interacts with people, tasks, technologies, and the work environment. The Introduction describes this as their proposed approach to AI in health care delivery. The supplied passages do not give page numbers. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, for a management decision tool built and deployed in a hospital in Utah for diabetes management, there was a challenge of not offering all the information that was desired by clinicians and patients to decide on type 2 diabetes management [5].”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Introduction
What the sources say
- Supportedcites [5] Tarumi, Leveraging artificial intelligence to improve chronic disease care: methods and application to pharmacotherapy decision support for type-2 diabetes mellitus, Methods Inf Med source
The Results (Integration with EHR) state that the diabetes decision-support system was integrated with the University of Utah EHR and available in its production system. The Discussion (Limitations) explicitly says the system did not provide all the information clinicians and patients wanted for T2DM management, including actual medication co-pay costs. (LLM, conf=0.95)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The Translational Evaluation of Healthcare AI framework checklist offers an alternative to the CASoF checklist for implementation”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Discussion > Principal Findings
What the sources say
- Supportedcites [3] Reddy, Evaluation framework to guide implementation of AI systems into healthcare settings, BMJ Health Care Inform source
The Results describe TEHAI as a checklist for evaluating healthcare AI and specify checks during development, deployment, and ongoing monitoring (Results > Adoption; page unavailable). The paper does not name CASoF, but the claim uses CASoF to frame TEHAI as another implementation checklist rather than attributing a direct comparison to the source. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“however, its lack of sociotechnical components, such as human-AI integration, culture and organization, and monitoring and support, which are essential for adoption and maximizing utility, is a drawback [3].”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Discussion > Principal Findings
What the sources say
- Partly supportedcites [3] Reddy, Evaluation framework to guide implementation of AI systems into healthcare settings, BMJ Health Care Inform sourceExperimental — accuracy not yet measured
The Results sections on Adoption and Utility describe workflow integration, real-world acceptance, contextual relevance, and ongoing monitoring as important to evaluating healthcare AI. They do not identify human–AI integration, organizational culture, or monitoring and support as the specific set of components claimed, nor say that all three are essential for maximizing utility. Page numbers are unavailable in our copy. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
13 more claims
Create a free account to see all 86 claims and every source verdict.
- Probably not a substantial claim. 56
- It may state two separate claims as one. 20
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“A total of 65 panelists were recruited: 21 (32%) doctors, 10 (15%) health care experts or researchers, 9 (12%) AI researchers”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Results > The Modified Delphi Study
Reported 12 · recomputed 13.846154
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“A total of 14 (65%) of these addressed specific areas in the AI development cycle”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Results > Literature Synthesis
Reported 65 · recomputed 70
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“A total of 65 panelists were recruited: 21 (32%) doctors, 10 (15%) health care experts or researchers, 9 (12%) AI researchers, 4 (6%) health informaticians, 4 (6%) nurses, and 18 (28%) other professionals.”
§ Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study > Results > The Modified Delphi Study
Reported 65 · recomputed 66
Citations
Cites
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388 · cited 1×
“…and TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) are examples of models that are narrow in their application and are focused on trials, performance, and comparison, which are only helpful in a single phase of the AI life cycle [8,9].”
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246 · cited 1×
“In general, AI tools have shown promise in development, but few have been able to translate into the real-world settings for patient management [4].”
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665 · cited 1×
“…goes for the DECIDE-AI (Developmental and Exploratory Clinical Investigations of Decision Support Systems Driven by Artificial Intelligence) checklist, which is focused on reporting studies that involve the evaluation of AI systems during their implementation phase in the clinical setting [36].”
Artificial Intelligence in Health Care: A Report From the National Academy of Medicine
Matheny, M. E., Whicher, D., & Thadaney Israni, S. (2020) · JAMA · doi:10.1001/jama.2019.21579 · Cited by 628 · cited 1×
“The National Academy of Medicine report highlighted the need to “understand the technical, cognitive, social, and political factors in play and incentives impacting integration of Artificial Intelligence (AI) into health care workflows” [2].”
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., Ashrafian, H., Beam, A. L., Chan, A.-W., Collins, G. S., Deeks, A. D. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30218-1 · Cited by 555
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
We Agree, Don’t We? The Delphi Method for Health Environments Research
Taylor, E. (2020) · HERD: Health Environments Research & Design Journal · doi:10.1177/1937586719887709 · Cited by 316 · cited 1×
“The Delphi method is a procedure for reaching a consensus with a group of people who are typically experts on the subject through controlled assessments [11].”
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287
Evaluation framework to guide implementation of AI systems into healthcare settings
Reddy, S., Rogers, W., Makinen, V.-P., Coiera, E., Brown, P., Wenzel, M., Weicken, E., Ansari, S., Mathur, P., Casey, A., et al. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100444 · Cited by 230 · cited 3×
“The literature search identified 20 studies [1,3,7,14-30] that proposed a framework, guideline, or approach for the design, development, implementation, or evaluation of AI for clinical use cases (Figure 2).”
Cited by
苗书英, Zhang, Z., & Bao, C. (2026) · Journal of Nursing Management · doi:10.1155/jonm/5288231 · Cited by 3
Anonymous (2025) · JMIRx Med · doi:10.2196/69869 · Cited by 1
Zaki, S. (2025) · JMIRx Med · doi:10.2196/70058 · Cited by 1
Anonymous (2025) · JMIRx Med · doi:10.2196/69595 · Cited by 1
Thompson, K. (2025) · JMIRx Med · doi:10.2196/69593 · Cited by 1
Anonymous (2025) · JMIRx Med · doi:10.2196/69870 · Cited by 1
Saripalli, S. (2025) · JMIRx Med · doi:10.2196/69594 · Cited by 1
Owoyemi, A., Osuchukwu, J., Salwei, M. E., et al. (2025) · JMIRx Med · doi:10.2196/69537
“Anonymous [1] The paper [2] presents the Clinical Artificial Intelligence (AI) Sociotechnical Framework (CASoF), a structured approach to guide the planning, design, development, and implementation of AI systems in health care settings.”
Self-citationNurfadhilah, E., Aini, L. R., Santosa, A., Uliniansyah, M. T., Gunarso, G., Putra, P. K., Pebiana, S., Fajri, R., Hidayati, N. N., Prastowo, R. D., et al. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7862-6.ch003
Olagoke, A. A., Aidoo-Frimpong, G. A., Adebayo, C. T., Shaw, J., Adegbile, O. E., Owoyemi, A., Qi, Z., & Okut, H. (2026) · Systems · doi:10.3390/systems14070739
Self-citationHuman Experience in AI-Enabled Workplace
Клєцова, Н., Klietsov, A., Pouransari, S., & Naji, R. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8282-1.ch010
Byberg, E. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1017322
Metadata
- Authors
- Owoyemi, A., Osuchukwu, J., Salwei, M. E., & Boyd, A.
- Affiliations
- University of Illinois Chicago (US)
- University of Cincinnati (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Owoyemi, Ayomide
- Venue
- JMIRx Med
- Publisher
- JMIR Publications Inc.
- Published
- 2025-02-20
- Volume
- 6
- Pages
- e65565-e65565
- ISSN
- 2563-6316 (ISSN-L)
- PMC
- PMC11867147
- OpenAlex
- W4407767410
- Semantic Scholar
- b022c8019ad98b128c2e174f3642deae9a1e7f54
- Cited by
- 18
- References
- 36
- Citations by year
- 2026: 8 · 2025: 9
- FWCI
- 3.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIRx Med (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Agency for Healthcare Research and Quality (K01 HS029042)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- artificial intelligence in healthcare, AI implementation, socio-technical factors, checklist development, AI lifecycle, clinical implementation of AI, healthcare professionals' perspectives, Delphi consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


