Assessing Barriers to Implementation of Machine Learning and Artificial Intelligence–Based Tools in Critical Care: Web-Based Survey Study
Open accessPublished version of a preprint
Abstract
Background: Although there is considerable interest in machine learning (ML) and artificial intelligence (AI) in critical care, the implementation of effective algorithms into practice has been limited. Objective: We sought to understand physician perspectives of a novel intubation prediction tool. Further, we sought to understand health care provider and nonprovider perspectives on the use of ML in health care. We aim to use the data gathered to elucidate implementation barriers and determinants of this intubation prediction tool, as well as ML/AI-based algorithms in critical care and health care in general. Methods: We developed 2 anonymous surveys in Qualtrics, 1 single-center survey distributed to 99 critical care physicians via email, and 1 social media survey distributed via Facebook and Twitter with branching logic to tailor questions for providers and nonproviders. The surveys included a mixture of categorical, Likert scale, and free-text items. Likert scale means with SD were reported from 1 to 5. We used student t tests to examine the differences between groups. In addition, Likert scale responses were converted into 3 categories, and percentage values were reported in order to demonstrate the distribution of responses. Qualitative free-text responses were reviewed by a member of the study team to determine validity, and content analysis was performed to determine common themes in responses. Results: Out of 99 critical care physicians, 47 (48%) completed the single-center survey. Perceived knowledge of ML was low with a mean Likert score of 2.4 out of 5 (SD 0.96), with 7.5% of respondents rating their knowledge as a 4 or 5. The willingness to use the ML-based algorithm was 3.32 out of 5 (SD 0.95), with 75% of respondents answering 3 out of 5. The social media survey had 770 total responses with 605 (79%) providers and 165 (21%) nonproviders. We found no difference in providers’ perceived knowledge based on level of experience in either survey. We found that nonproviders had significantly less perceived knowledge of ML (mean 3.04 out of 5, SD 1.53 vs mean 3.43, SD 0.941; P<.001) and comfort with ML (mean 3.28 out of 5, SD 1.02 vs mean 3.53, SD 0.935; P=.004) than providers. Free-text responses revealed multiple shared concerns, including accuracy/reliability, data bias, patient safety, and privacy/security risks. Conclusions: These data suggest that providers and nonproviders have positive perceptions of ML-based tools, and that a tool to predict the need for intubation would be of interest to critical care providers. There were many shared concerns about ML/AI in health care elucidated by the surveys. These results provide a baseline evaluation of implementation barriers and determinants of ML/AI-based tools that will be important in their optimal implementation and adoption in the critical care setting and health care in general.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inaccurate and underrepresentative training data sets for AI models can cause bias, misleading predictions, adverse events, and large-scale discrimination, causing barriers to adoption.
“Inaccurate and underrepresentative training data sets for AI models can cause bias, misleading predictions, adverse events, and large-scale discrimination, causing barriers to adoption [32,41,47,48].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [48]
The Discussion (Principal Findings; page not available) identifies concerns about systemic bias in data used by ML tools and patient safety as factors affecting confidence in implementation. The Conclusions also reports concerns about accuracy and reliability. The paper does not establish that inaccurate or underrepresentative training sets cause misleading predictions, adverse events, or large-scale discrimination; it reports perceived barriers rather than those outcomes. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPatient safety concerns causing adverse effects were noted.
“Patient safety concerns causing adverse effects were noted by Mlodzinski et al [48] and Vijayakumar et al [59].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [48]
The Results (Social Media Provider and Nonprovider Survey; page unavailable) list patient safety/outcomes among respondents’ concerns about ML and AI. The paper reports perceived risks, not adverse effects that occurred or were caused by those concerns. (LLM, conf=0.85)
Experimental — accuracy not yet measuredData security was identified as a major contributor toward hesitancy, with concerns around cybersecurity relating to both training and testing data as well as fear of trackers and spyware obtaining unsolicited data.
“Data security was identified as a major contributor toward hesitancy, with concerns around cybersecurity relating to both training and testing data as well as fear of trackers and spyware obtaining unsolicited data [39,41,42,45,48,77].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [48]
The Results (Social Media Provider and Nonprovider Survey; page not provided) identify privacy/security as a concern, and the Discussion (Principal Findings; page not provided) says it is a factor that needs addressing to build confidence. The paper does not specify cybersecurity risks involving training or testing data, or fears of trackers and spyware obtaining unsolicited data. Its survey results also do not establish data security as a major contributor to hesitancy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredSkepticism and a lack of understanding were seen as barriers to AI adoption.
“Skepticism and a lack of understanding were seen as barriers to AI adoption [48,51].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
Eysenbach, G. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.3.e34 · Cited by 6,480 · see citing works
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · see citing works
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167 · see citing works
Resistance to Medical Artificial Intelligence
Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019) · Journal of Consumer Research · doi:10.1093/jcr/ucz013 · Cited by 2,015 · see citing works
An interpretable machine learning model for accurate prediction of sepsis in the ICU
Nemati, S., Holder, A., Razmi, F., Stanley, M. D., Clifford, G. D., & Buchman, T. G. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000002936 · Cited by 832 · see citing works
Self-citationAn Index Combining Respiratory Rate and Oxygenation to Predict Outcome of Nasal High-Flow Therapy
Roca, O., Caralt, B., Messika, J., Samper, M. A., Sztrymf, B., Hernández, G., García-de-Acilu, M., Frat, J., Masclans, J. R., & Ricard, J.-D. (2018) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201803-0589oc · Cited by 813 · see citing works
Patient apprehensions about the use of artificial intelligence in healthcare
Richardson, J. P., Smith, C., Curtis, S., Watson, S., Zhu, X., Barry, B., & Sharp, R. R. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00509-1 · Cited by 424 · see citing works
Cognitive Processing of Self-Report Items in Educational Research: Do They Think What We Mean?
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., BLAZEVSKI, J., Bonney, C. R., de Groot, E., Gilbert, M. C., Musu, L. E., Kempler, T. M., et al. (2007) · Educational Psychologist · doi:10.1080/00461520701416231 · Cited by 385 · see citing works
Blease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12802 · Cited by 346 · see citing works
Patel, R. P., Gambrell, M., Speroff, T., Scott, T. A., Pun, B. T., Okahashi, J., Strength, C., Pandharipande, P. P., Girard, T. D., Burgess, H., et al. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e31819b8608 · Cited by 319 · see citing works
Nelson, C. A., Pérez-Chada, L. M., Creadore, A., Li, S. J., Lo, K., Manjaly, P., Pournamdari, A. B., Tkachenko, E., Barbieri, J. S., Ko, J. M., et al. (2020) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.5014 · Cited by 278 · see citing works
Bridging the implementation gap of machine learning in healthcare
Seneviratne, M. G., Shah, N. H., & Chu, L. (2020) · BMJ Innovations · doi:10.1136/bmjinnov-2019-000359 · Cited by 173 · see citing works
Ginestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · see citing works
Implementing machine learning in medicine
Verma, A. A., Murray, J., Greiner, R., Cohen, J., Shojania, K. G., Ghassemi, M., Straus, S. E., Pou-Prom, C., & Mamdani, M. (2021) · Canadian Medical Association Journal · doi:10.1503/cmaj.202434 · Cited by 144 · see citing works
Pehrson, L. M., Nielsen, M. B., & Lauridsen, C. A. (2019) · Diagnostics · doi:10.3390/diagnostics9010029 · Cited by 126 · see citing works
Himmelfarb, C. D., Méndez-Téllez, P. A., Wang, W., Pronovost, P. J., & Needham, D. M. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000287591.09487.70 · Cited by 75 · see citing works
Gallifant, J., Zhang, J., del Pilar Arias Lopez, M., Zhu, T., Camporota, L., Celi, L. A., & Formenti, F. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2021.09.025 · Cited by 72 · see citing works
Shashikumar, S. P., Wardi, G., Paul, P., Paul, P., Carlile, M., Brenner, L. N., Hibbert, K. A., North, C. M., Mukerji, S. S., Robbins, G. K., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.12.009 · Cited by 70 · see citing works
Self-citationYork, T. J., Jenney, H., & Jones, G. G. (2020) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100233 · Cited by 57 · see citing works
Machine Learning for Pulmonary and Critical Care Medicine: A Narrative Review
Mlodzinski, E., Stone, D. J., & Celi, L. A. (2020) · Pulmonary Therapy · doi:10.1007/s41030-020-00110-z · Cited by 46 · see citing works
Self-citationArtificial intelligence and the NHS: a qualitative exploration of the factors influencing adoption
Morrison, K. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2020-0258 · Cited by 43 · see citing works
A Twitter-based survey on marijuana concentrate use
Daniulaityte, R., Zatreh, M. Y., Lamy, F. R., et al. (2018) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2018.02.033 · Cited by 39 · see citing works
Wardi, G., Joel, I., Villar, J., Lava, M. S., Gross, E., Tolia, V. M., Seethala, R. R., Owens, R. L., Sell, R., Montesi, S. B., et al. (2019) · Journal of Intensive Care Medicine · doi:10.1177/0885066619871247 · Cited by 12 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Barriers to and facilitators of artificial intelligence adoption in health care: scoping review
Hassan, M., Kushniruk, A., & Borycki, E. (2024) · JMIR Human Factors · doi:10.2196/48633 · Cited by 324 · see citing works
Leading with AI in critical care nursing: challenges, opportunities, and the human factor
Hassan, E. A., & El-Ashry, A. M. (2024) · BMC Nursing · doi:10.1186/s12912-024-02363-4 · Cited by 72 · see citing works
21st Century critical care medicine: an overview
Padte, S., Samala Venkata, V., Mehta, P., Tawfeeq, S., Kashyap, R., & Surani, S. (2024) · World Journal of Critical Care Medicine · doi:10.5492/wjccm.v13.i1.90176 · Cited by 26 · see citing works
- Authors
- Mlodzinski, E., Wardi, G., Viglione, C., Nemati, S., Crotty Alexander, L., & Malhotra, A.
- Affiliations
- University of California San Diego (US)
- Corresponding author
- Mlodzinski, Eric
- ORCID
- Venue
- JMIR Perioperative Medicine
- Publisher
- JMIR Publications Inc.
- Published
- 2023-01-27
- Volume
- 6
- Pages
- e41056
- ISSN
- 2561-9128 (ISSN-L)
- PMC
- PMC10013679
- OpenAlex
- W4312846991
- Cited by
- 28
- References
- 24
- Citations by year
- 2026: 8 · 2025: 8 · 2024: 8 · 2023: 4
- FWCI
- 1.51
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- JMIR Perioperative Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Veterans Affairs (1I01BX004767)
- Gordon and Betty Moore Foundation (GBMF9052)
- Eli Lilly and Company
- LivaNova
- ResMed
- National Institutes of Health
- University of California, San Diego
- National Institute on Aging (R01AG063925)
- National Heart, Lung, and Blood Institute (R01HL148436, T32HL134632, R01HL147326, HL154026, R01HL157985, K24HL155884, R01HL154926)
- National Institute of General Medical Sciences (K23GM146092, R35GM143121)
- U.S. National Library of Medicine (R01LM013998)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- machine learning, artificial intelligence, implementation barriers, intensive care, intubation prediction, physician attitudes, willingness to use AI, healthcare workers, social media survey, technology adoption, clinical decision support, trust in AI, health equity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
