The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Artificial Intelligence and the Implementation Challenge
Open accessPublished version of a preprint
Abstract
BACKGROUND: Applications of artificial intelligence (AI) in health care have garnered much attention in recent years, but the implementation issues posed by AI have not been substantially addressed. OBJECTIVE: In this paper, we have focused on machine learning (ML) as a form of AI and have provided a framework for thinking about use cases of ML in health care. We have structured our discussion of challenges in the implementation of ML in comparison with other technologies using the framework of Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (NASSS). METHODS: After providing an overview of AI technology, we describe use cases of ML as falling into the categories of decision support and automation. We suggest these use cases apply to clinical, operational, and epidemiological tasks and that the primary function of ML in health care in the near term will be decision support. We then outline unique implementation issues posed by ML initiatives in the categories addressed by the NASSS framework, specifically including meaningful decision support, explainability, privacy, consent, algorithmic bias, security, scalability, the role of corporations, and the changing nature of health care work. RESULTS: Ultimately, we suggest that the future of ML in health care remains positive but uncertain, as support from patients, the public, and a wide range of health care stakeholders is necessary to enable its meaningful implementation. CONCLUSIONS: If the implementation science community is to facilitate the adoption of ML in ways that stand to generate widespread benefits, the issues raised in this paper will require substantial attention in the coming years.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, health care has seen slow success in the implementation of AI use cases.”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Clinicians expressed the need to understand both the scientific and clinical bases of the recommendations provided by the AI to confidently validate and apply the decision [19,20,36-38].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, successful utility requires successful adoption, and a number of studies have reported on the challenges encountered with implementing AI in health care [19-21].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [20]
In the citing paper
“Shaw et al. (2019) contend that decision support in health care can be useful only if its outputs are able to integrate with the human decision-making processes at the heart of health service delivery.”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Shaw et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 2×
“The value proposition of digital technology might be different for different stakeholder groups, and implementation frameworks direct attention to the implications of newly introduced technologies for patients, health care providers, managers, health policymakers, and others [8,25,26].”
BackgroundMethodDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Often, ML involves supervised methods that categorize data points, for example, as images of skin cancer or otherwise given datasets in which all data points (or at least a substantial subset) are associated with a label, ordinal, or category that is meant to be predicted or inferred [15].”
BackgroundNorth, D. C. (1991) · Journal of Economic Perspectives · doi:10.1257/jep.5.1.97 · Cited by 5,232
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837 · cited 1×
“We agree with a growing chorus of health care providers and researchers who suggest that ML will primarily serve to augment as opposed to replace the work of humans in the provision of health care in the near term [31], despite applications of automation in health care.”
BackgroundThe ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567 · cited 1×
“In cases where training data are partial or incomplete or only reflect a subset of a given population, the resulting model will only be relevant to the population of people represented in the dataset [47].”
BackgroundMaking sense of implementation theories, models and frameworks
Nilsen, P. (2015) · Implementation Science · doi:10.1186/s13012-015-0242-0 · Cited by 4,447 · cited 2×
“We have described use cases (and attendant value propositions) of ML in health care as more likely relating to decision support and less likely to automation, which begins to illustrate the implementation object of focus in ML initiatives [6,7].”
BackgroundMethodEvidence-Based Medicine: A New Approach to Teaching the Practice of Medicine
Guyatt, G. (1992) · JAMA · doi:10.1001/jama.1992.03490170092032 · Cited by 4,202
Machine Learning Applications in Cancer Prognosis and Prediction
Kourou, K., Exarchos, T. P., Exarchos, K. P., Karamouzis, M. V., & Fotiadis, D. I. (2015) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2014.11.005 · Cited by 3,338 · cited 1×
“Given these labels and the statistical models they help to train, ML can be very effective at determining the category in which any newly available individual data point belongs, thereby being useful in the effort to, for example, identify malignant cancers based on particular images [16].”
BackgroundBig other: Surveillance Capitalism and the Prospects of an Information Civilization
Zuboff, S. (2015) · Journal of Information Technology · doi:10.1057/jit.2015.5 · Cited by 3,205 · cited 1×
“As the innovations enabled by ML have taken on a more powerful role in driving global economies, corporations have strategically sought to acquire larger amounts of more diverse data to boost their capacity to develop ML algorithms [61].”
BackgroundInstitutions and Organizations: Ideas, Interests, and Identities
Scott, W. R. (2007) · Cited by 2,943
Research works printed but not linked: 8
- Ajay, A, 2019, The Economics of Artificial Intelligence: An Agenda
- 10.1016/j.socscimed.2012.11.021published · doi:10.1016/j.socscimed.2012.11.021published
- Agrawal, A, 2018, Prediction Machines: The Simple Economics of Artificial Intelligence
- RajpurkarPPolamreddiVBalakrishnanAResearchGate2017Cornell University LibraryMalaria likelihood prediction by effectively surveying households using deep reinforcement learning https://www.researchgate.net/publication/321324506_Malaria_Likelihood_Prediction_By_Effectively_Surveying_Households_Using_Deep_Reinforcement_Learning
- CaseyDBeyanOMurphyKFelzmannHRobot-Assisted Care for Elderly With Dementia: Is There a Potential for Genuine End-User Empowerment?Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts2015HRI'15 Extended AbstractsMarch 2-5, 2015Portland, OR2478
- RossiRAAhmedNKThe Network Data Repository with Interactive Graph Analytics and VisualizationProceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence2015AAAI'15January 25-30, 2015Austin Texas42923
- CulnaneCRubinsteinBTeagueVarXiv2017Health data in an open world https://arxiv.org/ftp/arxiv/papers/1712/1712.05627.pdf
- TsymbalAResearchGate2004Computer Science Department, Trinity College DublinThe problem of concept drift: definitions and related work https://www.researchgate.net/publication/228723141_The_Problem_of_Concept_Drift_Definitions_and_Related_Work
Cited by
Algorithmic bias: review, synthesis, and future research directions
Kordzadeh, N., & Ghasemaghaei, M. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1927212 · Cited by 770
“…explains the sources, consequences, and mitigation strategies in a range of domains from employment and people analytics (Gal et al., 2017) and healthcare (Shaw et al., 2019) to education (Williams et al., 2018), credit markets (Fuster et al., 2018), and criminal justice (Mulligan et al., 2019).”
Barriers to and facilitators of artificial intelligence adoption in health care: scoping review
Hassan, M., Kushniruk, A., & Borycki, E. (2024) · JMIR Human Factors · doi:10.2196/48633 · Cited by 435
“Clinicians expressed the need to understand both the scientific and clinical bases of the recommendations provided by the AI to confidently validate and apply the decision [19,20,36-38].”
BackgroundInfluentialPetersson, L., Larsson, I., Nygren, J. M., Nilsen, P., Neher, M., Reed, J. E., Tyskbo, D., & Svedberg, P. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08215-8 · Cited by 434
“AI is thus not one single type of technology but rather many different types within various application areas, e.g., diagnosis and treatment, patient engagement and adherence, and administrative activities [5, 6].”
BackgroundRole of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419
“Examples of key challenges that need to be addressed include data sharing and privacy issues, lack of algorithm transparency, the changing nature of health care work, financial concerns, and the demanding regulatory environment [1,3,9-13].”
BackgroundThe potential of artificial intelligence to improve patient safety: a scoping review
Bates, D. W., Levine, D., Syrowatka, A., Kuznetsova, M., Craig, K. J. T., Rui, A., Jackson, G. P., & Rhee, K. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00423-6 · Cited by 323
Metadata
- Authors
- Shaw, J., Rudzicz, F., Jamieson, T., & Goldfarb, A.
- Affiliations
- University of Toronto (CA)
- St. Michael's Hospital (CA)
- Corresponding author
- Shaw, James
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-07-10
- Volume
- 21
- Issue
- 7
- Pages
- e13659
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6652121
- OpenAlex
- W2947151330
- Cited by
- 389
- References
- 58
- Citations by year
- 2026: 62 · 2025: 70 · 2024: 68 · 2023: 77 · 2022: 49 · 2021: 36 · 2020: 22 · 2019: 5
- FWCI
- 8.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- artificial intelligence in healthcare, machine learning implementation, NASSS framework, clinical decision support, explainability, algorithmic bias, privacy and consent, healthcare workforce, scalability, technology adoption, implementation science
- MeSH terms
- Machine Learning, Artificial Intelligence, Humans, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial Intelligence and the Implementation Challenge. https://app.cochat.ai/papers/10.2196/13659
- The paper
- Shaw, J., Rudzicz, F., Jamieson, T., & Goldfarb, A. (2019). Artificial Intelligence and the Implementation Challenge. Journal of Medical Internet Research. https://doi.org/10.2196/13659
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.