CoChat
Sign Up
Cited by 252Artificial Intelligence in Healthcare and EducationElectronic Health Records SystemsEthics and Social Impacts of AI

Toward a responsible future: recommendations for AI-enabled clinical decision support

Open access
Labkoff, Steven, Oladimeji, Bilikis, Kannry, Joseph, Solomonides, Anthony, Leftwich, Russell, Koski, Eileen, Joseph, Amanda L, Lopez-Gonzalez, Monica +19 more·2024·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocae209PMC PMC11491642
Open this paper in CoChat — free account

Abstract

Background: Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging. Objectives: This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients. Materials and Methods: In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process. Results: Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided. Discussion: AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow. Conclusions: Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocae209

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The expert panel emphasized national-level monitoring and thorough documentation and training for end users.

    “The panel emphasized national-level monitoring (e.g., registries or mandatory reporting for AI tools) and thorough documentation and training for end users (Labkoff et al., 2024).”

    From An auditable and source-verified framework for clinical AI decision support: integrating retrieval-augmented generation with data provenance (2026) · cites this paper as Labkoff et al., 2024

    The Results and Recommendations sections call for national AI-CDS safety monitoring, a reporting clearinghouse, documentation, and comprehensive end-user training. The Monitoring and reporting section says the clearinghouse should standardize adverse-event reports, but does not recommend a registry or mandatory reporting for AI tools. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Organizational studies show that models co-designed by interdisciplinary teams achieve greater clinical accuracy and user acceptance compared to those developed in disciplinary silos.

    “Organizational studies show that models co-designed by interdisciplinary teams achieve greater clinical accuracy and user acceptance compared to those developed in disciplinary silos434.”

    From Artificial intelligence-driven multi-omics approaches in Alzheimer's disease: Progress, challenges, and future directions (2025) · cites this paper as [434]

    In Recommendations > Building safe and trustworthy systems (page unavailable), the paper recommends multidisciplinary involvement throughout AI development and says diverse teams could enhance model-development accuracy. It does not report organizational studies comparing interdisciplinary teams with disciplinary silos, or show greater clinical accuracy and user acceptance in such a comparison. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Labkoff et al. (2024) convened an expert panel that issued recommendations for AI-enabled CDS, including building safe and trustworthy systems and establishing robust validation, verification, and certification processes.

    “Labkoff et al. (2024) also convened an expert panel which issued recommendations for AI-enabled CDS. Key among them are building safe and trustworthy systems and establishing robust validation, verification, and certification processes.”

    From An auditable and source-verified framework for clinical AI decision support: integrating retrieval-augmented generation with data provenance (2026) · cites this paper as Labkoff et al., 2024

  • The expert panel's envisioned framework covers model training, explainability, continuous monitoring, and regulatory oversight-a lifecycle approach to AI safety.

    “Their envisioned framework covers model training, explainability, continuous monitoring, and regulatory oversight—essentially a lifecycle approach to AI safety (Labkoff et al., 2024).”

    From An auditable and source-verified framework for clinical AI decision support: integrating retrieval-augmented generation with data provenance (2026) · cites this paper as Labkoff et al., 2024

  • Emerging guidelines from experts and regulators emphasize that AI tools should be safe, validated, and continuously monitored.

    “emerging guidelines from experts and regulators emphasize that AI tools should be safe, validated, and continuously monitored (Labkoff et al., 2024).”

    From An auditable and source-verified framework for clinical AI decision support: integrating retrieval-augmented generation with data provenance (2026) · cites this paper as Labkoff et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [1]Collen MF. Computer medicine: its application today and tomorrow. Minn Med. 1966; 49(11): 1705- 1707. 5979482
  2. [2]Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform. 2016; 25(Suppl 1): S103- S116. 10.15265/IYS-2016-s034 PMC5171504 27488402 · DOI
  3. [3]Bleich HL. Computer-based consultation: electrolyte and acid-base disorders. Am J Med. 1972; 53(3): 285- 291. 4559984
  4. [4]Parker RC, Miller RA. Creation of realistic appearing simulated patient cases using the INTERNIST-1/QMR knowledge base and interrelationship properties of manifestations. Methods Inf Med. 1989; 28(4): 346- 351. 2695784
  5. [5]Barnett GO, Cimino JJ, Hupp JA, et al DXPlain: an evolving diagnostic decision-support system. JAMA. 1987; 258(1): 67- 74. 3295316
  6. [6]Lincoln MJ, Turner CW, Haug PJ, et al ILIAD training enhances medical students’ diagnostic skills. J Med Syst. 1991; 15(1): 93- 110. 1748852
  7. [8]Miller RA, Gardner RM. Recommendations for responsible monitoring and regulation of clinical software systems. J Am Med Inform Assoc. 1997; 4(6): 442- 457. 9391932 PMC61262
  8. [9]Labkoff SE, Sittig DF. Who watches the watchers: working towards safety for EHR knowledge resources. Appl Clin Inform. 2017; 8(2): 680- 685. 10.4338/ACI-2017-02-IE-0032 28657638 PMC6241753 · DOI
  9. [10]Bright TJ, Wong A, Dhurjati R, et al Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012; 157(1): 29- 43. 10.7326/0003-4819-157-1-201207030-00450 22751758 · DOI · Paper page
  10. [12]Advancing health care AI through ethics, evidence and equity. (n.d.). Accessed January 18, 2024. https://www.ama-assn.org/practice-management/digital/advancing-health-care-ai-through-ethics-evidence-and-equity
  11. [15]Solomonides AE, Koski E, Atabaki SM, et al Defining AMIA’s artificial intelligence principles. J Am Med Inform Assoc. 2022; 29(4): 585- 591. 10.1093/jamia/ocac006 35190824 PMC8922174 · DOI · Paper page
  12. [17]Shah NH, Halamka JD, Saria S, et al A nationwide network of health AI assurance laboratories. JAMA. 2024; 331(3): 245. 38117493
  13. [18]Stead WW, Aliferis C. Health AI assurance laboratories. JAMA. 2024; 331(12): 1061- 1062. 10.1001/jama.2024.1084 38345804 · DOI
  14. [19]Sittig DF, Boxwala A, Wright A, et al A lifecycle framework illustrates eight stages necessary for realizing the benefits of patient-centered clinical decision support. J Am Med Inform Assoc. 2023; 30(9): 1583- 1589. 10.1093/jamia/ocad122 PMID: 37414544; PMCID: PMC10436138. 37414544 PMC10436138 · DOI
  15. [20]Ferryman K, Mackintosh M, Ghassemi M. Considering biased data as informative artifacts in AI-assisted health care. N Engl J Med. 2023; 389(9): 833- 838. 10.1056/NEJMra2214964 37646680 · DOI
  16. [21]Alvarez JM, Colmenarejo AB, Elobaid A, et al Policy advice and best practices on bias and fairness in AI. Ethics Inf Technol. 2024; 26(2): 31. 10.1007/s10676-024-09746-w · DOI
  17. [22]Liu S, McCoy AB, Peterson JF, et al Leveraging explainable artificial intelligence to optimize clinical decision support. J Am Med Inform Assoc. 2024; 31(4): 968- 974. 38383050 PMC10990514
  18. [24]Wang L, Zhang Z, Wang D, et al Human-centered design and evaluation of AI-empowered clinical decision support systems: a systematic review. Front Comput Sci. 2023; 5: 1187299.
  19. [28]Software Engineering Models and Methods Course. (n.d.). Accessed January 16, 2024. https://www.computer.org/product/education/software-engineering-models-course
  20. [31]Shortliffe EH, Cimino JJ, Chiang MF, eds. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. 5th ed. Springer; 2021.
  21. [32]Finnell JT, Dixon BE, eds. Clinical Informatics Study Guide: Text and Review. 2nd ed. Springer; 2022.
  22. [33]Levy DR, Sloss EA, Chartash D, et al Reflections on the documentation burden reduction AMIA Plenary Session through the lens of 25 × 5. Appl Clin Inform. 2023; 14(1): 11- 15. 10.1055/a-1976-2052 36356593 PMC9812582 · DOI
  23. [34]Brender TD. Medicine in the era of artificial intelligence: Hey Chatbot, write me an H&P. JAMA Intern Med. 2023; 183(6): 507. 10.1001/jamainternmed.2023.1832 37115537 · DOI
  24. [36]Khan M, Bates D, Kovacheva V. (n.d.). The quest for equitable health care: the potential for artificial intelligence. Accessed January 10, 2024. https://catalyst.nejm.org/doi/full/10.1056/CAT.21.0293 · DOI
  25. [37]Ledford H. Millions of black people affected by racial bias in health-care algorithms. Nature. 2019; 574(7780): 608- 609. 10.1038/d41586-019-03228-6 31664201 · DOI · Paper page
  26. [38]Glauser W. AI in health care: improving outcomes or threatening equity? CMAJ. 2020; 192(1): E21- E22. 10.1503/cmaj.1095838 31907237 PMC6944301 · DOI
  27. [39]Nordling L. A fairer way forward for AI in health care. Nature. 2019; 573(7775): S103- S105. 10.1038/d41586-019-02872-2 31554993 · DOI · Paper page
  28. [40]Courtland R. Bias detectives: the researchers are striving to make algorithms fair. Nature. 2018; 558(7710): 357- 360. 10.1038/d41586-018-05469-3 29925973 · DOI · Paper page
  29. [43]Donald M, Berwick MD. MPP: the moral determinants of health. Accessed January 31, 2024. https://jamanetwork.com/journals/jama/article-abstract/2767353
  30. [44]Datasheets for datasets. (n.d.). Accessed January 21, 2024. https://cacm.acm.org/magazines/2021/12/256932-datasheets-for-datasets/abstract
  31. [48]Sendak MP, Gao M, Brajer N, Balu S. Presenting machine learning model information to clinical end users with model facts labels. NPJ Digit Med. 2020; 3(1): 41. 10.1038/s41746-020-0253-3 32219182 PMC7090057 · DOI · Paper page
  32. [49]Liu Y, Chen P-HC, Krause J, Peng L. How to read articles that use machine learning: users’ guides to the medical literature. JAMA. 2019; 322(18): 1806- 1816. 10.1001/jama.2019.16489 31714992 · DOI
  33. [51]Potnis KC, Ross JS, Aneja S, Gross CP, Richman IB. Artificial intelligence in breast cancer screening: evaluation of FDA device regulation and future recommendations. JAMA Intern Med. 2022; 182(12): 1306- 1312. 10.1001/jamainternmed.2022.4969 36342705 PMC10623674 · DOI
  34. [52]Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021; 27(4): 582- 584. 10.1038/s41591-021-01312-x 33820998 · DOI · Paper page
  35. [55]Parikh RB, Teeple S, Navathe AS. Addressing bias in artificial intelligence in health care. JAMA. 2019; 322(24): 2377- 2378. 10.1001/jama.2019.18058 31755905 · DOI · Paper page
  36. [59]Greenes RA, Del Fiol G., eds. Clinical Decision Support and Beyond: Progress and Opportunities in Knowledge-Enhanced Health and Healthcare. 3rd ed. Academic Press; 2023.
  37. [62]Finnell JT, Dixon BE, eds. Clinical Informatics Study Guide: Text and Review. Springer International Publishing; 2022. 10.1007/978-3-030-93765-2 · DOI
  38. [63]Bates DW, Auerbach A, Schulam P, Wright A, Saria S. Reporting and implementing interventions involving machine learning and artificial intelligence. Ann Intern Med. 2020; 172(11_Suppl): S137- S144. 10.7326/M19-0872 32479180 · DOI
  39. [66]Bates DW, Levine DM, Salmasian H, et al The safety of inpatient health care. N Engl J Med. 2023; 388(2): 142- 153. 36630622

Cited by

Sorted by citations

Authors
Labkoff, S., Oladimeji, B., Kannry, J., Solomonides, A., Leftwich, R., Koski, E., Joseph, A. L., Lopez-Gonzalez, M., Fleisher, L. A., Nolen, K., Dutta, S., Levy, D. R., Price, A., Barr, P. J., Hron, J. D., Lin, B., Srivastava, G., Pastor, N., Luque, U. S., Bui, T. T. T., Singh, R., Williams, T., Weiner, M. G., Naumann, T., Sittig, D. F., Jackson, G. P., & Quintana, Y.
Affiliations
  • Beth Israel Deaconess Medical Center (US)
  • UnitedHealth Group (United States) (US)
  • Icahn School of Medicine at Mount Sinai (US)
  • Endeavor Health (US)
  • Vanderbilt University (US)
  • IBM (United States) (US)
  • University of Victoria (CA)
  • Cognitive Insights for Artificial Intelligence , Baltimore, MD,
  • University of Pennsylvania (US)
  • Pfizer (United States) (US)
  • Harvard University (US)
  • Yale University (US)
  • Dartmouth College (US)
  • Boston Children's Hospital (US)
  • Vitalera , Barcelona 08028,
  • Massachusetts College of Pharmacy and Health Sciences (US)
  • American Medical Informatics Association (US)
  • Weill Cornell Medicine (US)
  • Microsoft (United States) (US)
  • The University of Texas Health Science Center at Houston (US)
  • Intuitive Surgical (United States) (US)
Corresponding author
Quintana, Yuri
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2024-09-26
Volume
31
Issue
11
Pages
2730-2739
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocae209
PMC
PMC11491642
OpenAlex
W4402858717
Cited by
252
References
6
Citations by year
2026: 165 · 2025: 83 · 2024: 4
FWCI
23.56
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
Funders
National Institute on Drug Abuse (P30DA029926)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
artificial intelligence in healthcare, clinical decision support, AI safety, trustworthiness, algorithmic bias, transparency, explainability, safety monitoring, adverse event reporting, regulatory oversight, responsible AI, data privacy, fairness, end-user training, consensus recommendations
MeSH terms
Artificial Intelligence, Humans, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review