CoChat
Sign Up
Cited by 330Artificial Intelligence in Healthcare and EducationRadiology practices and educationHealth Policy Implementation Science

Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors

Open access
Strohm, Lea, Hehakaya, Charisma, Ranschaert, Erik R., Boon, Wouter P. C., Moors, Ellen H. M.·2020·European Radiology
DOI 10.1007/s00330-020-06946-yPMC PMC7476917
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: The objective was to identify barriers and facilitators to the implementation of artificial intelligence (AI) applications in clinical radiology in The Netherlands. Materials and methods: Using an embedded multiple case study, an exploratory, qualitative research design was followed. Data collection consisted of 24 semi-structured interviews from seven Dutch hospitals. The analysis of barriers and facilitators was guided by the recently published Non-adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework for new medical technologies in healthcare organizations. Results: Among the most important facilitating factors for implementation were the following: (i) pressure for cost containment in the Dutch healthcare system, (ii) high expectations of AI’s potential added value, (iii) presence of hospital-wide innovation strategies, and (iv) presence of a “local champion.” Among the most prominent hindering factors were the following: (i) inconsistent technical performance of AI applications, (ii) unstructured implementation processes, (iii) uncertain added value for clinical practice of AI applications, and (iv) large variance in acceptance and trust of direct (the radiologists) and indirect (the referring clinicians) adopters. Conclusion: In order for AI applications to contribute to the improvement of the quality and efficiency of clinical radiology, implementation processes need to be carried out in a structured manner, thereby providing evidence on the clinical added value of AI applications. Key Points: • Successful implementation of AI in radiology requires collaboration between radiologists and referring clinicians. • Implementation of AI in radiology is facilitated by the presence of a local champion. • Evidence on the clinical added value of AI in radiology is needed for successful implementation.

AI snapshot

Written from the paper’s full text.

TL;DR

This qualitative multiple-case study in seven Dutch hospitals identifies facilitators and barriers to implementing AI applications in radiology using the NASSS framework. It finds that local champions, cost pressure, expected benefits, and innovation strategies facilitate implementation, while inconsistent technical performance, unstructured implementation, uncertain added value, and variable acceptance hinder it.

Key findings

  • Facilitating factors included expected added benefit for diagnostic practice (18 interviewees), expected operational benefits (17), easy integration into PACS (15), minimizing workflow changes (14), and local champions (10); hospital-wide innovation strategies were present in 4/7 hospitals and innovation managers in 3/7.
  • Hindering factors included uncertain funding (14 interviewees), inconsistent acceptance/trust among radiologists (14), inconsistent technical performance (10), doubting quality and safety (10), technical knowledge necessary (10), unstructured planning and monitoring (10), unstructured workflow implementation (9), and no empirical evidence on AI app validation (9).
  • Acceptance and trust varied among direct adopters (radiologists) and indirect adopters (referring clinicians); in three hospitals, referring clinicians did not trust the AI output and redid manual bone age analysis for every scan.
  • BoneXpert was the only AI application in clinical use across several Dutch hospitals; its smooth PACS integration was cited by 10 interviewees, but inconsistent acceptance of BoneXpert was also cited by 10.

Methods

  • Embedded multiple case study with an exploratory qualitative design, investigating seven Dutch hospitals and focusing on departments using BoneXpert, a commercial software-only medical device for automated bone maturity assessment from pediatric hand X-rays.
  • Data collection consisted of 24 semi-structured interviews from February to June 2019—20 case interviews across the seven hospitals plus 4 key informant interviews—complemented by document analysis; interviews lasted 20–80 min, 21 were face-to-face and 3 by telephone, and thematic saturation was reached.
  • Analysis was guided by the NASSS framework across seven domains; concepts from interviews were compared with the original framework, which was refined for AI in radiology, especially the adopter system with local champions, radiologists as direct adopters, and referring clinicians as indirect adopters.

Limitations

  • The paper states that the research focuses only on AI applications in Dutch radiology departments and cannot be generalized to other healthcare systems; varied interviewee roles and positions limit generalizability to other hospitals in The Netherlands and beyond.
  • The paper states that interviewees varied in experience and knowledge because AI implementation in radiology is at an early stage, so the sample may be biased toward individuals with particular interest and above-average positive attitudes toward AI.
  • The paper states that further research should examine applications with higher technical complexity than BoneXpert and representing a larger part of radiologists’ diagnostic work; evident from the text, the study relies on qualitative interviews and documents, with no quantitative measurement of clinical or organizational outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s00330-020-06946-y

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 32 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    Pressure on healthcare budgets stimulates Dutch hospitals and radiology departments to develop and implement innovative technologies that promise efficiency and/or quality gains.

    “First, pressure on healthcare budgets stimulates Dutch hospitals and radiology departments to develop and implement innovative technologies that promise efficiency and/or quality gains.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Radiologists, hospital management, and technology developers expect a large added value of AI applications in clinical practice, despite little empirical evidence.

    “although there is little empirical evidence, radiologists, members of hospital management, and technology developers expect a large added value of the AI applications in clinical practice.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For AI applications to be perceived as user-friendly by radiologists, they need to be easily integrated into existing IT systems such as PACSs.

    “in order for AI applications to be perceived as user-friendly by radiologists, they need to be easily integrated into existing IT systems used by radiologists, such as picture archiving and communication systems (PACSs).”
    Quote in the paperSubstantial claim
  • Findingown data

    A designated innovation manager was present in three of seven cases.

    “Such leadership is also manifested by the presence or absence of a designated innovation manager (present in three of the seven cases).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Interviewees mentioned the need to reframe their professional identity and responsibilities as a consequence of the arrival of AI applications.

    “interviewees mentioned the need to reframe their professional identity and responsibilities as a consequence of the arrival of AI applications.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 28
  • Classified as probably not a substantial claim. 13

  • May combine two claims.

    Findingown data

    Interviewees expect AI applications to provide operational benefits such as diminishing workloads, time-saving, more consistent reporting across radiologists, and advanced service availability.

    “The interviewees mentioned two main benefits: (1) improved diagnostic practice due to more precise and objective diagnoses, avoidance of mistakes, and the automation of cumbersome tasks; and (2) operational benefits, such as diminishing workloads, time-saving, more consistent reporting across radiologists, and advanced service availability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Interviewees expect AI applications to improve diagnostic practice through more precise and objective diagnoses, avoidance of mistakes, and automation of cumbersome tasks.

    “The interviewees mentioned two main benefits: (1) improved diagnostic practice due to more precise and objective diagnoses, avoidance of mistakes, and the automation of cumbersome tasks; and (2) operational benefits, such as diminishing workloads, time-saving, more consistent reporting across radiologists, and advanced service availability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Local champions are radiologists with a particularly strong interest in AI applications and usually better than average understanding of their technical aspects.

    “These local champions are radiologists that show a particularly strong interest in AI applications and usually have better than average understanding of the technical aspects of AI applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The benefits and costs of using AI may be unequally divided over departments, which complicates funding decisions.

    “the benefits and costs of using AI may be unequally divided over departments, which complicates funding decisions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Clinical benefits or organizational goals that might be achieved by using AI applications are not clearly established ex-ante and therefore hard to assess after implementation.

    “clinical benefits or organizational goals that might be achieved by using AI applications are not clearly established ex-ante and therefore hard to assess after implementation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    When evidence on technical performance is available, publications on validation of algorithms are based on laboratory rather than clinical settings.

    “When evidence on the technical performance is available (such as with BoneXpert), interviewees noted that publications on the validation of the algorithms are based on laboratory rather than clinical settings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Some radiologists doubt the quality and safety of an AI application and fail to adopt or abandon AI applications against a background of lacking technical understanding of AI.

    “Against a background of lacking technical understanding of AI, some radiologists are doubting the quality and safety of an application and fail to adopt or abandon AI applications.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    To overcome opposition from skeptical colleagues, local champions provide general information on AI and promote opportunities for experimentation with an application.

    “To overcome the opposition of potential skeptical colleagues, local champions appear to follow two strategies: (1) providing general information on AI or on an AI application in particular through scientific articles, books, and presentations; and (2) promoting opportunities for experimentation with an application, e.g., by organizing showcases or installing a test version of the application.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Findingreasoning

    AI applications should be implemented without large changes to routines and workflow practices, avoiding additional steps for reporting the AI result.

    “Also, AI applications should be implemented without large changes to routines and workflow practices, i.e., by avoiding additional steps for reporting the result of the AI application.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    While the possibility of AI replacing radiologists was extensively mentioned in recent radiology publications, none of the interviewees in this research identified with this threat.

    “While the possibility of AI replacing radiologists, and thereby threatening their professional identity, was extensively mentioned in many recent radiology publications [5, 9, 16, 44], none of the interviewees in this research identified with this threat.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [5] Sogani J, Allen B Jr, Dreyer K, McGinty G (2019) Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin Imaging 59(July 2019):8–11. https://doi.org/10.1016/j.clinimag.2019.08.001 source

      Not checked in the free run.

    • cites [9] Chockley K, Emanuel E (2016) The end of radiology? Three threats to the future practice of radiology. J Am Coll Radiol 13(12):1415–1420. https://doi.org/10.1016/j.jacr.2016.07.010 source

      Not checked in the free run.

    • cites [16] Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL (2018) Artificial intelligence in radiology. Nat Rev Cancer 18(8):500–510. https://doi.org/10.1038/s41568-018-0016-5 source

      Not checked in the free run.

    • cites [44] Nawrocki T, Maldjian PD, Slasky SE, Contractor SG (2018) Artificial intelligence and radiology: have rumors of the radiologist’s demise been greatly exaggerated? Acad Radiol 25(8):967–972. https://doi.org/10.1016/j.acra.2017.12.027 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Interviewees envision radiologists of the future as "imaging consultants" who play an active role in an interdisciplinary patient-focused hospital environment.

    “they envision radiologists of the future as “imaging consultants” who play an active role in an interdisciplinary patient-focused hospital environment [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Nederlandse Vereniging voor Radiologie (2016) De Rol van de Radioloog in 2020. Utrecht. https://www.radiologen.nl/system/files/bestanden/documenten/de_rol_van_de_radioloog_in_2020.pdf. Accessed 17 Nov 2019

      Not checked in the free run.

  • May combine two claims.

    Findingreasoning

    If other departments are to cover parts of the additional costs, they need to learn about the technology and potential benefits of using AI applications, which requires efficient communication between departments; this communication was a shortcoming in several studied cases.

    “If other departments are to cover parts of the additional costs, they need to learn about the technology and the potential benefits of using AI applications. This requires efficient communication between departments, a shortcoming in several of the studied cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    There is a lack of empirical evidence on the effect of AI applications on the radiological workflow and on their added value for clinical radiology practice.

    “there is a lack of empirical evidence on the effect of AI applications on the radiological workflow, as well as their added value for clinical radiology practice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In three hospitals, referring clinicians did not trust the output of the AI application and redid a manual bone age analysis for every scan.

    “in three hospitals, we found that the referring clinicians did not trust the output of the AI application and redid a manual bone age analysis for every scan.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Implementation plans do not specify how the AI application should be integrated into the clinical workflow, leading to significant variations in the way the application is used in different departments.

    “implementation plans do not specify how the AI application should be integrated into the clinical workflow, which leads to significant variations in the way the application is used in different departments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Local champions are vital in initiating and stimulating implementation within their department and taking the lead during the entire process.

    “interviewees and document analysis [28, 29] show that local champions are vital in initiating and stimulating implementation within their department and taking the lead during the entire process.”
    • Also stated in: § Conclusion · plain
      “One of the important facilitating factors is the presence of a “local champion,” an individual with a strong personal interest in AI applications who most often initiates and actively advances AI implementation in the organization.”
    • Also stated in: § Key Points · plain
      “Implementation of AI in radiology is facilitated by the presence of a local champion.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [28] Nederlandse Vereniging voor Radiologie (2016) De Rol van de Radioloog in 2020. Utrecht. https://www.radiologen.nl/system/files/bestanden/documenten/de_rol_van_de_radioloog_in_2020.pdf. Accessed 17 Nov 2019

      Not checked in the free run.

    • cites [29] Nederlandse Vereniging van Ziekenhuizen (2018) Ziekenhuiszorg in Cijfers 2018. Utrecht. https://ziekenhuiszorgincijfers.nl/assets/uploads/NVZ-Brancherapport-2018.pdf

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Interviewed users experienced the integration of BoneXpert into the PACS as very smooth, which was the main reason for its perceived user-friendliness.

    “the interviewed users experienced the integration of BoneXpert into the PACS as very smooth, being the main reason for its perceived user-friendliness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On radiology department level, only one hospital had a formalized innovation strategy regarding AI, while four more hospitals were developing such a strategic approach at the time of the research.

    “On radiology department-level, only one hospital had a formalized innovation strategy regarding AI. However, four more hospitals were developing such a strategic approach at the time of this research.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    BoneXpert is one of the first commercial applications of AI in radiology and appeared to be the only AI application in clinical use across several hospitals in The Netherlands.

    “BoneXpert is one of the first commercial applications of AI in radiology [25] and appeared to be the only AI application in clinical use across several hospitals in The Netherlands.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] Lee H, Tajmir S, Lee J et al (2017) Fully automated deep learning system for bone age assessment. J Digit Imaging 30(4):427–441. https://doi.org/10.1007/s10278-017-9955-8 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The Dutch radiology departments vary strongly in the number and types of internally available AI applications.

    “The Dutch radiology departments vary strongly in the number and types of internally available AI applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The Dutch healthcare system is confronted with a constant rise in demand accompanied by strong pressure to limit associated costs.

    “The Dutch healthcare system is confronted with a constant rise in demand accompanied by a strong pressure to limit associated costs [27].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] Ministerie van Volksgezondheid Welzijn en Sport (2018) Kamerbrief over Aanbieding Bestuurlijk Akkoord (Hoofdlijnenakkoord) Medisch-Specialistische Zorg 2019-2022. https://www.rijksoverheid.nl/documenten/kamerstukken/2018/06/04/kamerbrief-over-hoofdlijnenakkoord-medisch-specialistische-zorg-2019-2022. Accessed 21 Nov 2019

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Among the most prominent hindering factors is the uncertain added value for clinical practice, which causes low acceptance of AI applications among adopters and complicates the mobilization of funds to acquire AI applications.

    “Among the most prominent hindering factors is the uncertain added value for clinical practice, which causes low acceptance of AI applications among adopters and complicates the mobilization of funds to acquire AI applications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A large number of false negatives is more dangerous because a potential lesion might be overlooked.

    “Having a large number of false negatives is even more dangerous, because it means that a potential lesion might get overlooked.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Some technical understanding is imperative for quality and safety assessment and therefore creates trust in the AI application's reliability.

    “However, some technical understanding is imperative for quality and safety assessment and therefore create trust in the AI application’s reliability [8, 21, 30].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [8] Char DS, Shah NH, Magnus D (2018) Implementing machine learning in health care ’ addressing ethical challenges. N Engl J Med 378(11):981–983. https://doi.org/10.1056/NEJMp1714229 source

      Not checked in the free run.

    • cites [21] Recht MP, Dewey M, Dreyer K et al (2020) Integrating artificial intelligence into the clinical practice of radiology: challenges and recommendations. Eur Radiol:1–9. https://doi.org/10.1007/s00330-020-06672-5 source

      Not checked in the free run.

    • cites [30] Rubin DL (2019) Artificial intelligence in imaging: the radiologist’s role. J Am Coll Radiol 16(9):1309–1317. https://doi.org/10.1016/j.jacr.2019.05.036 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    Interviewees varied in related experience and knowledge due to the early stage of AI implementation, so the sample is possibly biased towards individuals with a particular interest and above-average positive attitude towards AI applications.

    “Interviewees varied with regard to their related experience and knowledge, due to the early stage of implementation of AI in radiology practice. Therefore, the sample of interviewees is possibly biased towards individuals with a particular interest and above-average positive attitude towards AI applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    In order for AI applications to contribute to improvement of quality and efficiency of clinical radiology, implementation processes need to be carried out in a structured manner, thereby providing evidence on clinical added value.

    “In order for AI applications to contribute to the improvement of the quality and efficiency of clinical radiology, implementation processes need to be carried out in a structured manner, thereby providing evidence on the clinical added value of AI applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Evidence on the clinical added value of AI in radiology is needed for successful implementation.

    “Evidence on the clinical added value of AI in radiology is needed for successful implementation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Technology implementation in hospital settings involves a large variety of stakeholders and organizational procedures, with strong routines and professional identities, as well as strict legal and regulatory standards.

    “Technology implementation in hospital settings involves a large variety of stakeholders and organizational procedures, with strong routines and professional identities, as well as strict legal and regulatory standards [18–20].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [18] Pope C, Halford S, Turnbull J, Prichard J, Calestani M, May C (2013) Using computer decision support systems in NHS emergency and urgent care: Ethnographic study using normalisation process theory. BMC Health Serv Res 13(1) source

      Not checked in the free run.

    • cites [19] Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O (2004) Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 82(4):581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x source

      Not checked in the free run.

    • cites [20] Greenhalgh T, Wherton J, Papoutsi C et al (2017) Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 19(11):e367. https://doi.org/10.2196/jmir.8775 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Literature has mainly blamed the failure of early CAD systems on low technical performance, while other potential barriers such as organizational or social aspects have been largely ignored.

    “Literature has mainly blamed this on low technical performance of these early applications [5, 14–16], while other potential barriers to successful implementation, such as organizational or social aspects, have been largely ignored [17].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [5] Sogani J, Allen B Jr, Dreyer K, McGinty G (2019) Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin Imaging 59(July 2019):8–11. https://doi.org/10.1016/j.clinimag.2019.08.001 source

      Not checked in the free run.

    • cites [14] van Ginneken B, Schaefer-Prokop CM, Prokop M (2011) Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology. 261(3):719–732. https://doi.org/10.1148/radiol.11091710 source

      Not checked in the free run.

    • cites [15] Kohli A, Jha S (2018) Why CAD failed in mammography. J Am Coll Radiol 15(3):535–537. https://doi.org/10.1016/j.jacr.2017.12.029 source

      Not checked in the free run.

    • cites [16] Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL (2018) Artificial intelligence in radiology. Nat Rev Cancer 18(8):500–510. https://doi.org/10.1038/s41568-018-0016-5 source

      Not checked in the free run.

    • cites [17] Nishikawa RM, Bae KT (2018) Importance of better human-computer interaction in the era of deep learning: mammography computer-aided diagnosis as a use case. J Am Coll Radiol 15(1):49–52. https://doi.org/10.1016/j.jacr.2017.08.027 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    AI applications in radiology are in an emerging phase, so it is too early to evaluate their implementation.

    “Considering that AI applications in radiology are in an emerging phase, it is too early to evaluate their implementation [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] Recht MP, Dewey M, Dreyer K et al (2020) Integrating artificial intelligence into the clinical practice of radiology: challenges and recommendations. Eur Radiol:1–9. https://doi.org/10.1007/s00330-020-06672-5 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    AI is expected to cause large changes in clinical work practices and requires complementary skills from radiologists.

    “AI is expected to cause large changes in clinical work practices and requires complementary skills from radiologists [5, 7, 8].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [5] Sogani J, Allen B Jr, Dreyer K, McGinty G (2019) Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin Imaging 59(July 2019):8–11. https://doi.org/10.1016/j.clinimag.2019.08.001 source

      Not checked in the free run.

    • cites [7] Jha S, Topol EJ (2016) Adapting to artificial intelligence: Radiologists and pathologists as information specialists. JAMA 316(22):2353–2354. https://doi.org/10.1001/jama.2016.17438 source

      Not checked in the free run.

    • cites [8] Char DS, Shah NH, Magnus D (2018) Implementing machine learning in health care ’ addressing ethical challenges. N Engl J Med 378(11):981–983. https://doi.org/10.1056/NEJMp1714229 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Interviews were conducted until thematic saturation was reached, meaning no new themes appeared during additional interviews.

    “Interviews were conducted until thematic saturation was reached, meaning when no new themes appeared during additional interviews.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The NASSS framework aims to detect determinants of implementation processes of complex technologies in healthcare on seven domains.

    “The NASSS framework aims to detect the determinants of implementation processes of complex technologies in healthcare on seven domains: the condition, technology, value proposition, the adopter system (patient, lay caregivers, individual technology user, and other staff), the organization, and the wider institutional and social context.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] Greenhalgh T, Wherton J, Papoutsi C et al (2017) Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 19(11):e367. https://doi.org/10.2196/jmir.8775 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Using maximum variability logic, interviewees occupying different positions within the participating hospitals were selected.

    “Using a maximum variability logic, interviewees occupying different positions within the participating hospitals were selected (see Table 1 and Appendix A).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The concepts identified in the interviews were compared with the original NASSS framework, which was afterwards refined.

    “The concepts identified in the interviews were compared with the original NASSS framework, which was afterwards refined.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Implementation of AI in clinical radiology practice is complex and has so far been slow.

    “While technical performance of AI applications is expected to increase continuously, its implementation in clinical radiology practice is rather complex and has so far been slow [6, 12, 13].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [6] He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K (2019) The practical implementation of artificial intelligence technologies in medicine. Nat Med 25(1):30–36. https://doi.org/10.1038/s41591-018-0307-0 source

      Not checked in the free run.

    • cites [12] Dreyer KJ, Geis JR (2017) When machines think: Radiology’s next frontier. Radiology. 285(3):713–718. https://doi.org/10.1148/radiol.2017171183 source

      Not checked in the free run.

    • cites [13] Yu KH, Kohane IS (2019) Framing the challenges of artificial intelligence in medicine. BMJ Qual Saf 28(3):238–241. https://doi.org/10.1136/bmjqs-2018-008551 source

      Not checked in the free run.

36 of 36

  • The authors state this as an assumption.

    AI applications will encounter barriers to implementation.

    “However, in view of the unsuccessful widespread diffusion of earlier CAD systems, it can be assumed that AI applications will encounter barriers to implementation.”

  • The paper relies on this without support.

    A large number of false positives create additional work for the radiologist.

    “In clinical terms, a large number of false positives create additional work for the radiologist, which was the case of earlier unsuccessful CAD applications.”

  • The paper relies on this without support.

    Currently, CE marks are granted without requiring proof of the performance and added benefit for clinical practice.

    “Currently, CE marks are granted without requiring proof of the performance and added benefit for clinical practice.”

  • The paper relies on this without support.

    The new MDR requires CE certification through a notified body and necessitates a large increase in requirements on quality, safety, and post-market surveillance.

    “The new MDR requires CE certification through a notified body and necessitates a large increase in requirements on quality, safety, and post-market surveillance.”

  • The paper relies on this without support.

    Measuring clinical and organizational benefits of AI at a micro-level is difficult, and there is no standard methodology to measure increases in the quality of diagnosis.

    “One reason is that measuring clinical and organizational benefits of AI on a micro-level is difficult. There is, for example, no standard methodology to measure increases in the quality of diagnosis.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Lack of accuracy of AI output is considered by clinicians and patients to pose a potential risk of harm.

    “The lack of accuracy of AI output was also considered to pose a potential risk of harm by both clinicians and patients, as, in some cases where the algorithm may output false negative results, it may provide an incorrect sense of reassurance and cause a delay in treatment.”

    From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [57]

    The Results section on hindering factors says clinicians doubted AI applications’ quality and safety and that false negatives could cause lesions to be overlooked. The paper does not report patients’ views or specifically say that false negatives create false reassurance and delay treatment; its interviewee list contains no patients. Page numbers are not available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • When an AI algorithm is too sensitive and provides false positive results, it may add work and costs to the treatment process.

    “However, in cases where the algorithm is too sensitive, thereby providing false positive results, it may add work and costs to the treatment process [39,42,57,61,68].”

    From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [57]

    In Results, under “Identified hindering factors for AI implementation in radiology” (page unavailable), the paper says that many false positives create additional work for radiologists. It does not say that false positives increase treatment-process costs; its discussion of healthcare costs concerns budget pressures more generally. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Clinicians expressed fear around having to reframe their professional identity and responsibilities.

    “Clinicians expressed fear around having to reframe their professional identity and responsibilities [57].”

    From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [57]

    The Results section says interviewees mentioned a need to reframe their professional identity and responsibilities, and separately notes that radiologists’ attitudes toward AI ranged to fear. It does not say clinicians expressed fear about that reframing; it specifically says none of the interviewees expressed fear of being replaced by AI. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The presence of AI experts, perhaps a multidisciplinary team with clinical scientists, data science and subject matter experts with AI skills, an innovation manager, AI experts to provide training, and local champions within departments involved in the end-to-end process, was considered an important element for adoption.

    “The presence of AI experts, perhaps a multidisciplinary team, particularly with clinical scientists, data science and subject matter experts with AI skills, an innovation manager, AI experts to provide training, and local champions within departments involved in the end-to-end process, was considered an important element for adoption [32,57,59,62,71].”

    From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [57]

    The Results section identifies local champions as vital to implementation throughout the process and describes a designated innovation manager as a sign of innovation leadership. It does not identify the proposed multidisciplinary team of clinical scientists, data scientists and subject-matter experts, or AI experts providing training, as adoption factors. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Lack of regulation and policies from the government, including uncertainty around legal direction or law, is a barrier to the application of AI technologies.

    “A lack of regulation and policies from the government, including uncertainty around legal direction or law, was presented as a barrier to the application of AI technologies [37,45,57,60,77,78].”

    From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [57]

    The Results section lists regulatory and legal uncertainties as a hindering factor for AI implementation in radiology and links them to a lack of jurisprudence under European regulations and unresolved legal responsibility. It does not say that government regulations or policies themselves are absent; it discusses uncertainty about how existing and forthcoming rules apply. Page numbers are not available in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reference 31 is described as a survey of radiology residents

    Minor

    The discussion cites reference 31 as evidence from radiology residents, while its listed title identifies its subjects as medical students.

  • Not all 24 interviews were from hospitals

    Minor

    The abstract attributes all 24 interviews to the seven hospitals, but Table 1 includes four key informants from professional organizations or an imaging technology provider.

  • Seven NASSS domains claimed, but six listed

    Minor

    The sentence says the framework covers seven domains but enumerates only six: condition, technology, value proposition, adopter system, organization, and wider context.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Tsang L, Kracov DA, Mulryne J et al (2017) The impact of artificial intelligence on medical innovation in the European Union and United States. Intellect Prop Technol Law J 29(8):3–12

Cited by

Sorted by citations

Related

Authors
Strohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C., & Moors, E. H. M.
Affiliations
  • Utrecht University (NL)
  • University Medical Center Utrecht (NL)
  • Elisabeth-TweeSteden Ziekenhuis (NL)
Corresponding author
Hehakaya, Charisma
ORCID
Venue
European Radiology
Publisher
Springer Science and Business Media LLC
Published
2020-05-26
Volume
30
Issue
10
Pages
5525-5532
ISSN
  • 0938-7994 (ISSN-L)
  • 1432-1084
DOI
10.1007/s00330-020-06946-y
PMC
PMC7476917
OpenAlex
W3032075591
Cited by
329
References
40
Citations by year
2026: 61 · 2025: 78 · 2024: 80 · 2023: 58 · 2022: 36 · 2021: 15 · 2020: 1
FWCI
8.10
Type
journal article
Language
en
Open access
Licence
Locations
  • European Radiology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Ghent University Academic Bibliography (Ghent University) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Funders
University Medical Center Utrecht
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
Keywords
artificial intelligence in radiology, NASSS framework, radiologists, referring clinicians, local champions, trust in AI, cost containment, Dutch healthcare system
MeSH terms
Radiologists, Artificial Intelligence, Data Collection, Humans, Netherlands, Radiography, Radiology, Program Evaluation, Program Development, Qualitative Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review