Acceptance, Barriers, and Facilitators to Implementing Artificial Intelligence–Based Decision Support Systems in Emergency Departments: Quantitative and Qualitative Evaluation
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Background: Despite the increasing availability of clinical decision support systems (CDSSs) and rising expectation for CDSSs based on artificial intelligence (AI), little is known about the acceptance of AI-based CDSS by physicians and its barriers and facilitators in emergency care settings. Objective: We aimed to evaluate the acceptance, barriers, and facilitators to implementing AI-based CDSSs in the emergency care setting through the opinions of physicians on our newly developed, real-time AI-based CDSS, which alerts ED physicians by predicting aortic dissection based on numeric and text information from medical charts, by using the Unified Theory of Acceptance and Use of Technology (UTAUT; for quantitative evaluation) and the Consolidated Framework for Implementation Research (CFIR; for qualitative evaluation) frameworks. Methods: This mixed methods study was performed from March to April 2021. Transitional year residents (n=6), emergency medicine residents (n=5), and emergency physicians (n=3) from two community, tertiary care hospitals in Japan were included. We first developed a real-time CDSS for predicting aortic dissection based on numeric and text information from medical charts (eg, chief complaints, medical history, vital signs) with natural language processing. This system was deployed on the internet, and the participants used the system with clinical vignettes of model cases. Participants were then involved in a mixed methods evaluation consisting of a UTAUT-based questionnaire with a 5-point Likert scale (quantitative) and a CFIR-based semistructured interview (qualitative). Cronbach α was calculated as a reliability estimate for UTAUT subconstructs. Interviews were sampled, transcribed, and analyzed using the MaxQDA software. The framework analysis approach was used during the study to determine the relevance of the CFIR constructs. Results: All 14 participants completed the questionnaires and interviews. Quantitative analysis revealed generally positive responses for user acceptance with all scores above the neutral score of 3.0. In addition, the mixed methods analysis identified two significant barriers (System Performance, Compatibility) and two major facilitators (Evidence Strength, Design Quality) for implementation of AI-based CDSSs in emergency care settings. Conclusions: Our mixed methods evaluation based on theoretically grounded frameworks revealed the acceptance, barriers, and facilitators of implementation of AI-based CDSS. Although the concern of system failure and overtrusting of the system could be barriers to implementation, the locality of the system and designing an intuitive user interface could likely facilitate the use of optimal AI-based CDSS. Alleviating and resolving these factors should be key to achieving good user acceptance of AI-based CDSS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tong, A., Sainsbury, P., & Craig, J. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm042 · Cited by 42,509
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
Making sense of cronbach’s alpha.
Tavakol, M., & Dennick, R. (2011) · International Journal of Medical Education · doi:10.5116/ijme.4dfb.8dfd · Cited by 15,126
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 4×
“Because this model does not consider the unique characteristics of the clinical setting (eg, limited time and human resources in the emergency department setting) [18], we further adopted qualitative research techniques based on the CFIR to further identify barriers and facilitators of the AI-based CDSS [13].”
BackgroundMethodInfluentialGale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-117 · Cited by 10,865
Qualitative data analysis for applied policy research
Ritchie, J., & Spencer, L. (2010) · Analyzing qualitative data · doi:10.4324/9780203413081_chapter_9 · Cited by 4,800
Deo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 3×
“Despite the increased availability of CDSSs, including AI-based CDSSs in emergency care settings, the use of these systems is limited and has yet to achieve widespread implementation [1,9].”
BackgroundRodríguez-Pérez, R., & Bajorath, J. (2020) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-020-00314-0 · Cited by 718
Cited by
Rony, M. K. K., Kayesh, I., Bala, S. D., Akter, F., & Parvin, M. R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25718 · Cited by 190
“Regarding this, AI ’ s integration demands a united endeavor to overcome technical complexities and entrenched resistance within healthcare systems [50].”
BackgroundTrust in and Acceptance of Artificial Intelligence Applications in Medicine: Mixed Methods Study
Shevtsova, D., Ahmed, A., Boot, I. W. A., Sanges, C., Hudecek, M., Jacobs, J. J. L., Hort, S., & Vrijhoef, H. J. M. (2024) · JMIR Human Factors · doi:10.2196/47031 · Cited by 124
Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Rony, M. K. K., Numan, S. M., Akter, K., Tushar, H., Debnath, M., Johra, F. T., Akter, F., Mondal, S., Das, M., Uddin, M. J., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36702 · Cited by 93
“Fujimori et al. (2022) highlighted the role of established ethical principles in shaping AI development and deployment [61].”
BackgroundLiao, X., Yao, C., Jin, F., Zhang, J., & Liu, L. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-084398 · Cited by 50
Steffny, L., Dahlem, N., Reichl, L., Gisa, K., Greff, T., & Werth, D. (2023) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia230090 · Cited by 8
Arends, B. K. O., McCormick, J. M., van der Harst, P., Heus, P., & van Es, R. (2025) · European Journal of Clinical Investigation · doi:10.1111/eci.14387 · Cited by 5
“14,15,21 The payment model of the healthcare system may also play a role, and in the Netherlands, where healthcare costs are tightly regulated and shared between insurers and the government, financial aspects may not be as immediate a concern for clinicians.”
Metadata
- Authors
- Fujimori, R., Liu, K., Soeno, S., Naraba, H., Ogura, K., Hara, K., Sonoo, T., Ogura, T., Nakamura, K., & Goto, T.
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-06-13
- Volume
- 6
- Issue
- 6
- Pages
- e36501
- ISSN
- 2561-326X (ISSN-L)
- OpenAlex
- W4229366741
- Cited by
- 87
- References
- 13
- FWCI
- 3.77
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.