Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors

In the paper

“Literature has mainly blamed this on low technical performance of these early applications [5, 14–16], while other potential barriers to successful implementation, such as organizational or social aspects, have been largely ignored [17].”

What the sources say

• Not checked yetcites [5] Sogani J, Allen B Jr, Dreyer K, McGinty G (2019) Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin Imaging 59(July 2019):8–11. https://doi.org/10.1016/j.clinimag.2019.08.001 source
• Not checked yetcites [14] van Ginneken B, Schaefer-Prokop CM, Prokop M (2011) Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology. 261(3):719–732. https://doi.org/10.1148/radiol.11091710 source
• Not checked yetcites [15] Kohli A, Jha S (2018) Why CAD failed in mammography. J Am Coll Radiol 15(3):535–537. https://doi.org/10.1016/j.jacr.2017.12.029 source
• Partly supported

  In “Impact on oncology imaging > Detection” (page unavailable), the paper says traditional CADe systems had questionable accuracy and attributes their limited clinical impact in part to subhuman performance. It does not establish that the literature mainly blamed technical performance. The sentence cites [17] separately for the claim about organizational and social barriers, so that part is not attributed to this source. (LLM, conf=0.82)

  cites [16] Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL (2018) Artificial intelligence in radiology. Nat Rev Cancer 18(8):500–510. https://doi.org/10.1038/s41568-018-0016-5 sourceExperimental — accuracy not yet measured
• Partly supported

  In “CAD in Current Clinical Breast Imaging” (page unavailable), the paper argues that CAD implementation, rather than algorithm accuracy, has limited its clinical success. It also calls for attention to radiologists’ behavior, but does not establish that organizational or social barriers have been largely ignored across the literature. (LLM, conf=0.82)

  cites [17] Nishikawa RM, Bae KT (2018) Importance of better human-computer interaction in the era of deep learning: mammography computer-aided diagnosis as a use case. J Am Coll Radiol 15(1):49–52. https://doi.org/10.1016/j.jacr.2017.08.027 sourceExperimental — accuracy not yet measured

In the paper

“The interviewees mentioned two main benefits: (1) improved diagnostic practice due to more precise and objective diagnoses, avoidance of mistakes, and the automation of cumbersome tasks; and (2) operational benefits, such as diminishing workloads, time-saving, more consistent reporting across radiologists, and advanced service availability.”

In the paper

“The interviewees mentioned two main benefits: (1) improved diagnostic practice due to more precise and objective diagnoses, avoidance of mistakes, and the automation of cumbersome tasks; and (2) operational benefits, such as diminishing workloads, time-saving, more consistent reporting across radiologists, and advanced service availability.”

In the paper

“These local champions are radiologists that show a particularly strong interest in AI applications and usually have better than average understanding of the technical aspects of AI applications.”

In the paper

“the benefits and costs of using AI may be unequally divided over departments, which complicates funding decisions.”

In the paper

“clinical benefits or organizational goals that might be achieved by using AI applications are not clearly established ex-ante and therefore hard to assess after implementation.”

In the paper

“When evidence on the technical performance is available (such as with BoneXpert), interviewees noted that publications on the validation of the algorithms are based on laboratory rather than clinical settings.”

In the paper

“Against a background of lacking technical understanding of AI, some radiologists are doubting the quality and safety of an application and fail to adopt or abandon AI applications.”

In the paper

“To overcome the opposition of potential skeptical colleagues, local champions appear to follow two strategies: (1) providing general information on AI or on an AI application in particular through scientific articles, books, and presentations; and (2) promoting opportunities for experimentation with an application, e.g., by organizing showcases or installing a test version of the application.”

In the paper

“Also, AI applications should be implemented without large changes to routines and workflow practices, i.e., by avoiding additional steps for reporting the result of the AI application.”

In the paper

“While the possibility of AI replacing radiologists, and thereby threatening their professional identity, was extensively mentioned in many recent radiology publications [5, 9, 16, 44], none of the interviewees in this research identified with this threat.”

What the sources say

• Not checked yetcites [5] Sogani J, Allen B Jr, Dreyer K, McGinty G (2019) Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin Imaging 59(July 2019):8–11. https://doi.org/10.1016/j.clinimag.2019.08.001 source
• Not checked yetcites [9] Chockley K, Emanuel E (2016) The end of radiology? Three threats to the future practice of radiology. J Am Coll Radiol 13(12):1415–1420. https://doi.org/10.1016/j.jacr.2016.07.010 source
• Not checked yetcites [16] Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL (2018) Artificial intelligence in radiology. Nat Rev Cancer 18(8):500–510. https://doi.org/10.1038/s41568-018-0016-5 source
• Not checked yetcites [44] Nawrocki T, Maldjian PD, Slasky SE, Contractor SG (2018) Artificial intelligence and radiology: have rumors of the radiologist’s demise been greatly exaggerated? Acad Radiol 25(8):967–972. https://doi.org/10.1016/j.acra.2017.12.027 source

In the paper

“they envision radiologists of the future as “imaging consultants” who play an active role in an interdisciplinary patient-focused hospital environment [28].”

What the sources say

• Not checked yetcites [28] Nederlandse Vereniging voor Radiologie (2016) De Rol van de Radioloog in 2020. Utrecht. https://www.radiologen.nl/system/files/bestanden/documenten/de_rol_van_de_radioloog_in_2020.pdf. Accessed 17 Nov 2019

In the paper

“If other departments are to cover parts of the additional costs, they need to learn about the technology and the potential benefits of using AI applications. This requires efficient communication between departments, a shortcoming in several of the studied cases.”

In the paper

“there is a lack of empirical evidence on the effect of AI applications on the radiological workflow, as well as their added value for clinical radiology practice.”

In the paper

“in three hospitals, we found that the referring clinicians did not trust the output of the AI application and redid a manual bone age analysis for every scan.”

In the paper

“implementation plans do not specify how the AI application should be integrated into the clinical workflow, which leads to significant variations in the way the application is used in different departments.”

In the paper

“interviewees and document analysis [28, 29] show that local champions are vital in initiating and stimulating implementation within their department and taking the lead during the entire process.”

• Also stated in: § Conclusion · plain

  “One of the important facilitating factors is the presence of a “local champion,” an individual with a strong personal interest in AI applications who most often initiates and actively advances AI implementation in the organization.”

• Also stated in: § Key Points · plain

  “Implementation of AI in radiology is facilitated by the presence of a local champion.”

What the sources say

• Not checked yetcites [28] Nederlandse Vereniging voor Radiologie (2016) De Rol van de Radioloog in 2020. Utrecht. https://www.radiologen.nl/system/files/bestanden/documenten/de_rol_van_de_radioloog_in_2020.pdf. Accessed 17 Nov 2019
• Not checked yetcites [29] Nederlandse Vereniging van Ziekenhuizen (2018) Ziekenhuiszorg in Cijfers 2018. Utrecht. https://ziekenhuiszorgincijfers.nl/assets/uploads/NVZ-Brancherapport-2018.pdf

In the paper

“the interviewed users experienced the integration of BoneXpert into the PACS as very smooth, being the main reason for its perceived user-friendliness.”

In the paper

“On radiology department-level, only one hospital had a formalized innovation strategy regarding AI. However, four more hospitals were developing such a strategic approach at the time of this research.”

In the paper

“BoneXpert is one of the first commercial applications of AI in radiology [25] and appeared to be the only AI application in clinical use across several hospitals in The Netherlands.”

What the sources say

• Not checked yetcites [25] Lee H, Tajmir S, Lee J et al (2017) Fully automated deep learning system for bone age assessment. J Digit Imaging 30(4):427–441. https://doi.org/10.1007/s10278-017-9955-8 source

In the paper

“The Dutch radiology departments vary strongly in the number and types of internally available AI applications.”

In the paper

“The Dutch healthcare system is confronted with a constant rise in demand accompanied by a strong pressure to limit associated costs [27].”

What the sources say

• Not checked yetcites [27] Ministerie van Volksgezondheid Welzijn en Sport (2018) Kamerbrief over Aanbieding Bestuurlijk Akkoord (Hoofdlijnenakkoord) Medisch-Specialistische Zorg 2019-2022. https://www.rijksoverheid.nl/documenten/kamerstukken/2018/06/04/kamerbrief-over-hoofdlijnenakkoord-medisch-specialistische-zorg-2019-2022. Accessed 21 Nov 2019

In the paper

“Among the most prominent hindering factors is the uncertain added value for clinical practice, which causes low acceptance of AI applications among adopters and complicates the mobilization of funds to acquire AI applications.”

In the paper

“Having a large number of false negatives is even more dangerous, because it means that a potential lesion might get overlooked.”

In the paper

“However, some technical understanding is imperative for quality and safety assessment and therefore create trust in the AI application’s reliability [8, 21, 30].”

What the sources say

• Not checked yetcites [8] Char DS, Shah NH, Magnus D (2018) Implementing machine learning in health care ’ addressing ethical challenges. N Engl J Med 378(11):981–983. https://doi.org/10.1056/NEJMp1714229 source
• Not checked yetcites [21] Recht MP, Dewey M, Dreyer K et al (2020) Integrating artificial intelligence into the clinical practice of radiology: challenges and recommendations. Eur Radiol:1–9. https://doi.org/10.1007/s00330-020-06672-5 source
• Not checked yetcites [30] Rubin DL (2019) Artificial intelligence in imaging: the radiologist’s role. J Am Coll Radiol 16(9):1309–1317. https://doi.org/10.1016/j.jacr.2019.05.036 source

In the paper

“Interviewees varied with regard to their related experience and knowledge, due to the early stage of implementation of AI in radiology practice. Therefore, the sample of interviewees is possibly biased towards individuals with a particular interest and above-average positive attitude towards AI applications.”

In the paper

“In order for AI applications to contribute to the improvement of the quality and efficiency of clinical radiology, implementation processes need to be carried out in a structured manner, thereby providing evidence on the clinical added value of AI applications.”

In the paper

“Evidence on the clinical added value of AI in radiology is needed for successful implementation.”

In the paper

“Technology implementation in hospital settings involves a large variety of stakeholders and organizational procedures, with strong routines and professional identities, as well as strict legal and regulatory standards [18–20].”

What the sources say

• Not checked yetcites [18] Pope C, Halford S, Turnbull J, Prichard J, Calestani M, May C (2013) Using computer decision support systems in NHS emergency and urgent care: Ethnographic study using normalisation process theory. BMC Health Serv Res 13(1) source
• Not checked yetcites [19] Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O (2004) Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 82(4):581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x source
• Not checked yetcites [20] Greenhalgh T, Wherton J, Papoutsi C et al (2017) Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 19(11):e367. https://doi.org/10.2196/jmir.8775 source

In the paper

“Considering that AI applications in radiology are in an emerging phase, it is too early to evaluate their implementation [21].”

What the sources say

• Not checked yetcites [21] Recht MP, Dewey M, Dreyer K et al (2020) Integrating artificial intelligence into the clinical practice of radiology: challenges and recommendations. Eur Radiol:1–9. https://doi.org/10.1007/s00330-020-06672-5 source

In the paper

“AI is expected to cause large changes in clinical work practices and requires complementary skills from radiologists [5, 7, 8].”

What the sources say

• Not checked yetcites [5] Sogani J, Allen B Jr, Dreyer K, McGinty G (2019) Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin Imaging 59(July 2019):8–11. https://doi.org/10.1016/j.clinimag.2019.08.001 source
• Not checked yetcites [7] Jha S, Topol EJ (2016) Adapting to artificial intelligence: Radiologists and pathologists as information specialists. JAMA 316(22):2353–2354. https://doi.org/10.1001/jama.2016.17438 source
• Not checked yetcites [8] Char DS, Shah NH, Magnus D (2018) Implementing machine learning in health care ’ addressing ethical challenges. N Engl J Med 378(11):981–983. https://doi.org/10.1056/NEJMp1714229 source

In the paper

“Interviews were conducted until thematic saturation was reached, meaning when no new themes appeared during additional interviews.”

In the paper

“The NASSS framework aims to detect the determinants of implementation processes of complex technologies in healthcare on seven domains: the condition, technology, value proposition, the adopter system (patient, lay caregivers, individual technology user, and other staff), the organization, and the wider institutional and social context.”

What the sources say

• Not checked yetcites [20] Greenhalgh T, Wherton J, Papoutsi C et al (2017) Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 19(11):e367. https://doi.org/10.2196/jmir.8775 source

In the paper

“Using a maximum variability logic, interviewees occupying different positions within the participating hospitals were selected (see Table 1 and Appendix A).”

In the paper

“The concepts identified in the interviews were compared with the original NASSS framework, which was afterwards refined.”

In the paper

“While technical performance of AI applications is expected to increase continuously, its implementation in clinical radiology practice is rather complex and has so far been slow [6, 12, 13].”

What the sources say

• Not checked yetcites [6] He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K (2019) The practical implementation of artificial intelligence technologies in medicine. Nat Med 25(1):30–36. https://doi.org/10.1038/s41591-018-0307-0 source
• Not checked yetcites [12] Dreyer KJ, Geis JR (2017) When machines think: Radiology’s next frontier. Radiology. 285(3):713–718. https://doi.org/10.1148/radiol.2017171183 source
• Not checked yetcites [13] Yu KH, Kohane IS (2019) Framing the challenges of artificial intelligence in medicine. BMJ Qual Saf 28(3):238–241. https://doi.org/10.1136/bmjqs-2018-008551 source