From Code to Bedside: Implementing Artificial Intelligence Using Quality Improvement Methods
Open accessAbstract
Despite increasing interest in how artificial intelligence (AI) can augment and improve healthcare delivery, the development of new AI models continues to outpace adoption in existing healthcare processes. Integration is difficult because current approaches separate the development of AI models from the complex healthcare environments in which they are intended to function, resulting in models developed without a clear and compelling use case and not tested or scalable in a clinical setting. We propose that current approaches and traditional research methods do not support successful AI implementation in healthcare and outline a repeatable mixed-methods approach, along with several examples, that facilitates uptake of AI technologies into human-driven healthcare processes. Unlike traditional research, these methods do not seek to control for variation, but rather understand it to learn how a technology will function in practice coupled with user-centered design techniques. This approach, leveraging design thinking and quality improvement methods, aims to increase the adoption of AI in healthcare and prompt further study to understand which methods are most successful for AI implementations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This dual approach enables healthcare organizations to refine the technology, prioritizing user satisfaction and feedback.
“This dual approach enables healthcare organizations to refine the technology, prioritizing user satisfaction and feedback [57].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [57]
Repeated extensive user experience testing is recommended to arrive at a good design.
“To arrive at a good design, repeated extensive user experience testing is recommended11,18.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [11]
This type of study consists of repeated live clinical tests in which variation is key to understanding the functionality of the technology and workflow.
“This type of study consists of repeated live clinical tests in which variation is key to understanding the functionality of the technology and workflow11,18.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [11]
By adhering to the plan-do-study-adjust process, adjustments can be made frequently and rapidly to optimize the workflow.
“By adhering to the ‘plan, do, study, adjust’ process, adjustments can be made frequently and rapidly to optimize the workflow11,18.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [11]
The literature strongly recommends stating the necessary conditions for deployment before proceeding with implementation.
“The literature strongly recommends to state the necessary conditions for deployment before proceeding with the implementation11,19,20,26,87.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · see citing works
Porter, M. E. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1011024 · Cited by 5,084 · see citing works
From Triple to Quadruple Aim: Care of the Patient Requires Care of the Provider
Bodenheimer, T., & Sinsky, C. (2014) · The Annals of Family Medicine · doi:10.1370/afm.1713 · Cited by 3,462 · see citing works
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · see citing works
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · see citing works
The role of project management in achieving project success
Munns, A. K., & Bjeirmi, B. F. (1996) · International Journal of Project Management · doi:10.1016/0263-7863(95)00057-7 · Cited by 915 · see citing works
A design thinking framework for healthcare management and innovation
Roberts, J. P., Fisher, T. R., Trowbridge, M. J., & Bent, C. (2016) · Healthcare · doi:10.1016/j.hjdsi.2015.12.002 · Cited by 518 · see citing works
Understanding contexts by being there: case studies in bodystorming
Oulasvirta, A., Kurvinen, E., & Kankainen, T. (2003) · Personal and Ubiquitous Computing · doi:10.1007/s00779-003-0238-7 · Cited by 344 · see citing works
Study Designs for PDSA Quality Improvement Research
Speroff, T., & O’Connor, G. T. (2004) · Quality Management in Health Care · doi:10.1097/00019514-200401000-00002 · Cited by 252 · see citing works
Ten Ways Artificial Intelligence Will Transform Primary Care
Lin, S. Y., Mahoney, M. R., & Sinsky, C. A. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-05035-1 · Cited by 248 · see citing works
Self-citationArtificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril
Series, T. L. H. S., & National Academy of Medicine (2019) · doi:10.17226/27111 · Cited by 227 · see citing works
The Quality of Medical Evidence: Implications for Quality of Care
Eddy, D. M., & Billings, J. (1988) · Health Affairs · doi:10.1377/hlthaff.7.1.19 · Cited by 191 · see citing works
Using clinical simulation to study how to improve quality and safety in healthcare
Lamé, G., & Dixon‐Woods, M. (2018) · BMJ Simulation & Technology Enhanced Learning · doi:10.1136/bmjstel-2018-000370 · Cited by 180 · see citing works
New mindset in scientific method in the health field: Design Thinking
Ferreira, F. K., Song, E. H., Gomes, H. C., Garcia, É. B., & Ferreira, L. M. (2015) · Clinics · doi:10.6061/clinics/2015(12)01 · Cited by 76 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Davenport TH, Ronanki R. Artificial Intelligence for the Real World. Harv Bus Rev. 2018;96(1):108-116.
Cited by
Sorted by citations
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
Use of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160 · see citing works
“This dual approach enables healthcare organizations to refine the technology, prioritizing user satisfaction and feedback [57].”
BackgroundThe integration of AI in nursing: addressing current applications, challenges, and future directions
Wei, Q., Pan, S., Liu, X., Hong, M., Nong, C., & Zhang, W. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1545420 · Cited by 143 · see citing works
Advancing primary care with Artificial Intelligence and Machine Learning
Yang, Z., Silcox, C., Sendak, M., Rose, S., Rehkopf, D., Phillips, R., Peterson, L., Marino, M., Maier, J., Lin, S., et al. (2022) · Healthcare · doi:10.1016/j.hjdsi.2021.100594 · Cited by 38 · see citing works
Self-citationKhan, S. D., Hoodbhoy, Z., Raja, M. H. R., Kim, J. Y., Hogg, H. D. J., Manji, A. A. A., Gulamali, F., Hasan, A., Shaikh, A., Tajuddin, S., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000514 · Cited by 36 · see citing works
- Authors
- Smith, M., Sattler, A., Hong, G., & Lin, S.
- Affiliations
- Stanford Health Care (US)
- Corresponding author
- Lin, Steven
- Venue
- Journal of General Internal Medicine
- Published
- 2021-01-19
- Volume
- 36
- Issue
- 4
- Pages
- 1061-1066
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC8041947
- OpenAlex
- W3121857082
- Cited by
- 51
- References
- 14
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 10 · 2023: 8 · 2022: 4 · 2021: 4
- FWCI
- 1.89
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- AI implementation, health technology adoption, quality improvement methods, design thinking, user-centered design, mixed methods, clinical workflow integration, clinical implementation of AI
- MeSH terms
- Artificial Intelligence, Delivery of Health Care, Humans, Research Design, Quality Improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
