Clinical use of artificial intelligence requires AI-capable organizations
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
Artificial intelligence-based algorithms are being widely implemented in health care, even as evidence is emerging of bias in their design, problems with implementation, and potential harm to patients. To achieve the promise of using of AI-based tools to improve health, healthcare organizations will need to be AI-capable, with internal and external systems functioning in tandem to ensure the safe, ethical, and effective use of AI-based tools. Ideas are starting to emerge about the organizational routines, competencies, resources, and infrastructures that will be required for safe and effective deployment of AI in health care, but there has been little empirical research. Infrastructures that provide legal and regulatory guidance for managers, clinician competencies for the safe and effective use of AI-based tools, and learner-centric resources such as clear AI documentation and local health ecosystem impact reviews can help drive continuous improvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Regular maintenance is also necessary to update the model as new data becomes available and to retrain it if its performance drops [56, 62].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A related study highlighted that successful implementation of AI in healthcare requires organizations to be “AI-capable”, with appropriate infrastructures, resource allocation, and competency-building for health professionals [178].”
From Opportunities, challenges, and requirements for Artificial Intelligence (AI) implementation in Primary Health Care (PHC): a systematic review (2025) · cites this paper as [178]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Evolutionary Theory of Economic Change.
Loasby, B. J., Nelson, R. R., & Winter, S. G. (1983) · The Economic Journal · doi:10.2307/2232409 · Cited by 21,366
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Understanding dynamic capabilities
Winter, S. G. (2003) · Strategic Management Journal · doi:10.1002/smj.318 · Cited by 5,315 · cited 1×
“Scholarship in organizational sciences has framed organizational capabilities in terms of management’s ability to leverage resources and competencies to create routines(6) that can accomplish the work required to produce the desired outcome.(7) In addition, infrastructures(8) exist beyond the organization that can bot…”
BackgroundStructural racism and health inequities in the USA: evidence and interventions
Bailey, Z. D., Krieger, N., Agénor, M., Graves, J., Linos, N., & Bassett, M. T. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30569-x · Cited by 5,203
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
Reconceptualizing Organizational Routines as a Source of Flexibility and Change
Feldman, M. S., & Pentland, B. T. (2003) · Administrative Science Quarterly · doi:10.2307/3556620 · Cited by 4,020 · cited 1×
“They also serve as a vehicle for change, providing pathways for changes to propagate across an organization.(31) Routines standardize practices, reducing uncertainty and enabling measurement.”
BackgroundPredicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279
Gluesing, J. C. (2012) · Leadership in Science and Technology: A Reference Handbook · doi:10.4135/9781412994231.n14 · Cited by 3,210
(2009) · Choice Reviews Online · doi:10.5860/choice.46-6156 · Cited by 3,071
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887
The “All of Us” Research Program
The All of Us Research Program Investigators (2019) · New England Journal of Medicine · doi:10.1056/nejmsr1809937 · Cited by 2,440
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857
The Nature and Dynamics of Organizational Capabilities
(2001) · doi:10.1093/0199248540.001.0001 · Cited by 1,534
Ethical and legal challenges of artificial intelligence-driven healthcare
Gerke, S., Minssen, T., & Cohen, G. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00012-5 · Cited by 1,170
Sittig, D. F., & Singh, H. (2010) · Quality and Safety in Health Care · doi:10.1136/qshc.2010.042085 · Cited by 897
Organizational routines as a unit of analysis
Pentland, B. T. (2005) · Industrial and Corporate Change · doi:10.1093/icc/dth070 · Cited by 895
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855
The “All of Us” Research Program
(2019) · New England Journal of Medicine · doi:10.1056/nejmc1912496 · Cited by 843
Artificial Intelligence in Health Care: A Report From the National Academy of Medicine
Matheny, M. E., Whicher, D., & Thadaney Israni, S. (2020) · JAMA · doi:10.1001/jama.2019.21579 · Cited by 628
Medical Education Must Move From the Information Age to the Age of Artificial Intelligence
Wartman, S. A., & Combs, C. D. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002044 · Cited by 585
On the interpretability of artificial intelligence in radiology: challenges and opportunities
Reyes, M., Meier, R., Pereira, S., Silva, C. A., Dahlweid, F.-M., Tengg-Kobligk, H. V., Summers, R. M., & Wiest, R. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190043 · Cited by 484
Introduction: An Agenda for Infrastructure Studies
University of Michigan, Edwards, P., Bowker, G., University of Pittsburgh, Jackson, S., University of Michigan, Williams, R., & University of Edinburgh (2009) · Journal of the Association for Information Systems · doi:10.17705/1jais.00200 · Cited by 470
Artificial intelligence in medical education
Masters, K. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1595557 · Cited by 469
Artificial Intelligence in Health Care: Bibliometric Analysis
Guo, Y., Hao, Z., Zhao, S., Gong, J., & Yang, F. (2020) · Journal of Medical Internet Research · doi:10.2196/18228 · Cited by 461
Beyond Routines as Things: Introduction to the Special Issue on Routine Dynamics
Feldman, M. S., Pentland, B. T., D’Adderio, L., & Lazaric, N. (2016) · Organization Science · doi:10.1287/orsc.2016.1070 · Cited by 444
Sapci, A. H., & Sapci, H. A. (2020) · JMIR Medical Education · doi:10.2196/19285 · Cited by 377
Competencies for the Use of Artificial Intelligence–Based Tools by Health Care Professionals
Russell, R. G., Lovett Novak, L., Patel, M., Garvey, K. V., Craig, K. J. T., Jackson, G. P., Moore, D., & Miller, B. M. (2023) · Academic Medicine · doi:10.1097/acm.0000000000004963 · Cited by 320 · cited 1×
“In response to this gap, we subsequently conducted a series of semistructured interviews with subject matter experts in health professions education and healthcare AI, aiming to generate a set of competencies needed by frontline clinicians to evaluate and safely use AI-based tools.(49) The expertise of the participant…”
MethodSelf-citationMisrepresenting Race — The Role of Medical Schools in Propagating Physician Bias
Amutah, C., Greenidge, K., Mante, A., Munyikwa, M., Surya, S. L., Higginbotham, E., Jones, D. S., Lavizzo-Mourey, R., Roberts, D., Tsai, J., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmms2025768 · Cited by 317
Presenting machine learning model information to clinical end users with model facts labels
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0253-3 · Cited by 279 · cited 2×
“Scholars have begun to work on mechanisms and approaches to this challenge.(52) These Model Facts Labels should be structured to allow continuous learning for each of the target audiences, and ideally would be required for regulatory approval.”
BackgroundMastering the digital transformation through organizational capabilities: A conceptual framework
Konopik, J., Jahn, C., Schuster, T., Hoßbach, N., & Pflaum, A. (2022) · Digital Business · doi:10.1016/j.digbus.2021.100019 · Cited by 266
Atlas of AI: power, politics, and the planetary costs of artificial intelligence.
Can, M. (2022) · International Affairs · doi:10.1093/ia/iiac034 · Cited by 241
Accuracy of Race, Ethnicity, and Language Preference in an Electronic Health Record
Klinger, E. V., Carlini, S. V., Gonzalez, I., Hubert, S. S., Linder, J. A., Rigotti, N. A., Kontos, E. Z., Park, E. R., Marinacci, L. X., & Haas, J. S. (2015) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3102-8 · Cited by 234
Medicine and the Computer: The Promise and Problems of Change
Schwartz, W. B. (1970) · New England Journal of Medicine · doi:10.1056/nejm197012032832305 · Cited by 227
Regulation of predictive analytics in medicine
Parikh, R. B., Obermeyer, Z., & Navathe, A. S. (2019) · Science · doi:10.1126/science.aaw0029 · Cited by 223 · cited 1×
“are beginning to operate through the FDA and with support from professional organizations.(53,64) These will require continual investment and iterative improvements to keep pace with AI-based tool development and deployment in health care.”
BackgroundA Path for Translation of Machine Learning Products into Healthcare Delivery
(2020) · EMJ Innovations · doi:10.33590/emjinnov/19-00172 · Cited by 172
Wiljer, D., & Hakim, Z. (2019) · Journal of Medical Imaging and Radiation Sciences · doi:10.1016/j.jmir.2019.09.010 · Cited by 170
Neuroscience and behavioral genetics in US criminal law: an empirical analysis
Farahany, N. A. (2016) · Journal of Law and the Biosciences · doi:10.1093/jlb/lsv059 · Cited by 168
Diagnosing and Treating Systemic Racism
Evans, M. K., Rosenbaum, L., Malina, D., Morrissey, S., & Rubin, E. J. (2020) · New England Journal of Medicine · doi:10.1056/nejme2021693 · Cited by 152
Evaluating artificial intelligence in medicine: phases of clinical research
Park, Y., Jackson, G. P., Foreman, M. A., Gruen, D., Hu, J., & Das, A. K. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa033 · Cited by 150 · cited 1×
“Establishing and continuously improving the necessary routines will require allocation of organizational resources, including the development and support of competent personnel.(35) These routines can be implemented optimally by using existing organizational structures and routines (eg, regulatory and quality improvem…”
BackgroundArtificial Intelligence for Health Professions Educators
Lomis, K., Jeffries, P., Palatta, A., Sage, M., Sheikh, J., Sheperis, C., & Whelan, A. (2021) · NAM Perspectives · doi:10.31478/202109a · Cited by 136
A nonparametric updating method to correct clinical prediction model drift
Davis, S. E., Greevy, R. A., Fonnesbeck, C., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz127 · Cited by 110
Role of routines in collaborative work in healthcare organisations
Greenhalgh, T. (2008) · BMJ · doi:10.1136/bmj.a2448 · Cited by 109
Case Studies in Social Medicine — Attending to Structural Forces in Clinical Practice
Stonington, S. D., Holmes, S. M., Hansen, H., Greene, J. A., Wailoo, K. A., Malina, D., Morrissey, S., Farmer, P. E., & Marmot, M. G. (2018) · New England Journal of Medicine · doi:10.1056/nejmms1814262 · Cited by 107
Biomedical Informatics: Changing What Physicians Need to Know and How They Learn
Stead, W. W., Searle, J. R., Fessler, H., Smith, J. W., & Shortliffe, E. H. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181f41e8c · Cited by 103
The case for algorithmic stewardship for artificial intelligence and machine learning technologies
Eaneff, S., Obermeyer, Z., & Butte, A. J. (2020) · JAMA · doi:10.1001/jama.2020.9371 · Cited by 101
Embi, P. J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.4622 · Cited by 99
Watson, J., Hutyra, C. A., Clancy, S. M., Chandiramani, A., Bedoya, A., Ilangovan, K., Nderitu, N., & Poon, E. G. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooz046 · Cited by 95
Mediating the intersections of organizational routines during the introduction of a health IT system
Novak, L. L., Brooks, J. M., Gadd, C. S., Anders, S. H., & Lorenzi, N. M. (2012) · European Journal of Information Systems · doi:10.1057/ejis.2012.2 · Cited by 86
Self-citationArtificial Intelligence and Clinical Decision Making: The New Nature of Medical Uncertainty
Harish, V., Morgado, F., Stern, A. D., & Das, S. (2021) · Academic Medicine · doi:10.1097/acm.0000000000003707 · Cited by 73
Petersen, C. C., Smith, J. R. L., Freimuth, R. R., Goodman, K. W., Jackson, G. P., Kannry, J. L., Liu, H., Madhavan, S., Sittig, D. F., & Wright, A. T. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa319 · Cited by 73
Tolsgaard, M. G., Boscardin, C., Park, Y. S., Cuddy, M. M., & Sebok‐Syer, S. S. (2020) · Advances in Health Sciences Education · doi:10.1007/s10459-020-10009-8 · Cited by 68
Garvey, K. V., Thomas Craig, K. J., Russell, R., Novak, L. L., Moore, D., & Miller, B. M. (2022) · JMIR Medical Informatics · doi:10.2196/37478 · Cited by 62 · cited 1×
“To identify the competencies that currently guide the preparation of clinicians as they test, provide requirements for, and implement new AI-based technologies in practice settings, the authors completed a scoping review and large gaps in knowledge and training resources were identified.(47,48) Although many publicati…”
MethodSelf-citationMachine Learning: The Next Paradigm Shift in Medical Education
James, C. A., Wheelock, K. M., & Woolliscroft, J. O. (2021) · Academic Medicine · doi:10.1097/acm.0000000000003943 · Cited by 58
Kashyap, S., Morse, K. E., Patel, B., & Shah, N. H. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab154 · Cited by 43
Health information technology and dynamic capabilities
Leung, R. (2012) · Health Care Management Review · doi:10.1097/hmr.0b013e31823c9b55 · Cited by 37
Ones and zeros: Medical education and theory in the age of intelligent machines
Hodges, B. D. (2020) · Medical Education · doi:10.1111/medu.14149 · Cited by 22
Limited evidence for use of a black race modifier in eGFR calculations: a systematic review
Marzinke, M. A., Greene, D. N., Bossuyt, P. M., Chambliss, A. B., Cirrincione, L. R., McCudden, C. R., Melanson, S. E. F., Noguez, J. H., Patel, K., Radix, A. E., et al. (2022) · Clinical Chemistry · doi:10.1093/clinchem/hvab279 · Cited by 19
Garvey, K. V., Craig, K. J. T., Russell, R. G., Novak, L. L., Moore, D. E., Preininger, A. M., Jackson, G. P., & Miller, B. M. (2021) · Medical Science Educator · doi:10.1007/s40670-021-01377-w · Cited by 13
Self-citationA Tale of Two Doctors — Structural Inequalities and the Culture of Medicine
Aronson, L. (2017) · New England Journal of Medicine · doi:10.1056/nejmms1702140 · Cited by 4
Eisenhardt, K. M., & Martin, J. A. (2000) · Strategic Management Journal · doi:10.1002/1097-0266(200010/11)21:10/11
Research works printed but not linked: 3
- Davis, 1993, User acceptance of information technology: system characteristics, user perceptions and behavioural impacts, Int J Hum Comput Stud, 38, 475
- Coalition for Health AI, 2022, Blueprint for Trustworthy AI: Implementation Guidance and Assurance for Healthcare
- March, 1958, Organizations
Cited by
Reddy, S. (2024) · Implementation Science · doi:10.1186/s13012-024-01357-9 · Cited by 554
“It requires careful planning and execution [56].”
BackgroundMethodInfluentialYim, D., Khuntia, J., Parameswaran, V., & Meyers, A. (2024) · JMIR Medical Informatics · doi:10.2196/52073 · Cited by 118
“GenAI is costly to implement and integrate across all aspects of a health system [174].”
BackgroundZaleski, A. L., Berkowsky, R. S., Craig, K. J. T., & Pescatello, L. S. (2024) · JMIR Medical Education · doi:10.2196/51308 · Cited by 95
“As AI technologies have the potential to impact health disparities, it is essential to carefully evaluate their use to ensure inclusivity and appropriate messaging across demographics [27,52-54].”
BackgroundFazakarley, C.-A., Breen, M. A., Leeson, P., Thompson, B. T., & Williamson, V. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076950 · Cited by 67
“31 34 When discussing approaches to improve the acceptability of AI tools, participating professionals highlighted the importance of transparency between AI developers and their intended users.”
BackgroundYousefi, F., Dehnavieh, R., Laberge, M., Gagnon, M.-P., Ghaemi, M. M., Nadali, M., & Azizi, N. (2025) · BMC Primary Care · doi:10.1186/s12875-025-02785-2 · Cited by 70
“A related study highlighted that successful implementation of AI in healthcare requires organizations to be “AI-capable”, with appropriate infrastructures, resource allocation, and competency-building for health professionals [178].”
Metadata
- Authors
- Novak, L. L., Russell, R. G., Garvey, K., Patel, M., Thomas Craig, K. J., Snowdon, J., & Miller, B.
- Affiliations
- Vanderbilt University Medical Center (US)
- Vanderbilt University (US)
- Western Reserve Hospital (US)
- Clinical Evidence Development, Aetna®, Medical Affairs CVS Health® , Wellesley, Massachusetts, USA
- IBM (United States) (US)
- Corresponding author
- Novak, Laurie Lovett
- ORCID
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-04-06
- Volume
- 6
- Issue
- 2
- Pages
- ooad028-ooad028
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC10155810
- OpenAlex
- W4368377190
- Cited by
- 43
- References
- 64
- Citations by year
- 2026: 14 · 2025: 16 · 2024: 12 · 2023: 1
- FWCI
- 0.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- IBM Watson Health
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- artificial intelligence in healthcare, organizational capabilities, AI implementation, algorithmic bias, patient safety, AI ethics, clinician competence, AI governance, regulatory guidelines, AI documentation, organizational infrastructure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
