Implementing machine learning in medicine
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
[See related articles at www.cmaj.ca/lookup/doi/10.1503/cmaj.202066][1] and [www.cmaj.ca/lookup/doi/10.1503/cmaj.210036][2] KEY POINTS Machine learning — the process of developing systems that learn from data to recognize patterns and make accurate predictions of future events[1][3] — has
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Michie, S., van Stralen, M. M., & West, R. (2011) · Implementation Science · doi:10.1186/1748-5908-6-42 · Cited by 13,697
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Lost in knowledge translation: Time for a map?
Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., & Robinson, N. (2006) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.47 · Cited by 4,705 · cited 1×
“A number of excellent frameworks support data analytics and quality-improvement initiatives, including the Cross-Industry Standard Process for Data Mining (CRISP-DM), 7 the Model for Improvement developed by the Institute for Healthcare Improvement 8 and the Knowledge to Action 9 framework.”
MethodSelf-citationBig Data and Machine Learning in Health Care
Beam, A. L., & Kohane, I. S. (2018) · JAMA · doi:10.1001/jama.2017.18391 · Cited by 2,506
Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors
Koppel, R. (2005) · JAMA · doi:10.1001/jama.293.10.1197 · Cited by 2,126
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
Taylor, M. J., McNicholas, C., Nicolay, C., Darzi, A., Bell, D., & Reed, J. E. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2013-001862 · Cited by 1,871
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842
Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 1×
“The quality of input data can be assessed for completeness, correctness, concordance, plausibility and currency 18 through relatively simple, automated approaches and targeted manual validation.”
MethodArtificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
Asan, O., Bayrak, A. E., & Choudhury, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15154 · Cited by 825
Sendelbach, S., & Funk, M. (2013) · AACN Advanced Critical Care · doi:10.4037/nci.0b013e3182a903f9 · Cited by 497
Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration
Escobar, G. J., Liu, V. X., Schuler, A., Lawson, B., Greene, J. D., & Kipnis, P. (2020) · New England Journal of Medicine · doi:10.1056/nejmsa2001090 · Cited by 314 · cited 1×
“17 How should machine-learned solutions be designed?”
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294 · cited 1×
“21 In the case example (Box 1), the solution involved a prediction model, a communication system to convey patient risk to clinicians, and a clinical care pathway for high-risk patients.”
BackgroundA primer on PDSA: executing plan-do-study-act cycles in practice, not just in name
Leis, J. A., & Shojania, K. G. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-006245 · Cited by 235
Self-citationPhansalkar, S., Edworthy, J., Hellier, E., Seger, D. L., Schedlbauer, A., Avery, A. J., & Bates, D. W. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005264 · Cited by 226 · cited 1×
“In the case example, based on human factors principles, 22 a simple 3-level approach was selected to present actionable information to clinicians, with patients stratified into high, medium-and low-risk groups.”
MethodThe impact of machine learning on patient care: A systematic review
Ben-Israel, D., Jacobs, W. B., Casha, S., Lang, S., Ryu, W. H. A., de Lotbiniere-Bassett, M., & Cadotte, D. W. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101785 · Cited by 207
Verma, A. A., Guo, Y., Kwan, J. L., Lapointe-Shaw, L., Rawal, S., Tang, T., Weinerman, A., Cram, P., Dhalla, I. A., Hwang, S. W., et al. (2017) · CMAJ Open · doi:10.9778/cmajo.20170097 · Cited by 145 · cited 1×
“14 The hospital has a well-established critical care response team, staffed by a respiratory therapist, ICU nurse and ICU physician, which can be called by ward teams to urgently assess inpatients who may require transfer to the ICU.”
BackgroundSelf-citationZheng, K., Ratwani, R. M., & Adler-Milstein, J. (2020) · Annals of Internal Medicine · doi:10.7326/m19-0871 · Cited by 105 · cited 2×
“24,27 These can then be mapped to effective strategies to optimize implementation using approaches such as the Capability, Opportunity, Motivation, Behaviour (COM-B) model.”
BackgroundMethod
Cited by
Marwaha, J. S., Landman, A. B., Brat, G. A., Dunn, T., & Gordon, W. J. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00557-1 · Cited by 241
“Existing frameworks for digital health implementation are dif fi cult to operationalize for these dynamic organizations as they do not clearly de fi ne which stakeholders to involve, their roles, or the sequence of recommended steps 15 .”
MethodImplementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Machine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52
“66 Achieving this can be facilitated by priming stakeholders before introduction, 67 offering intuitive interfaces, providing continuous education, implementing in phases (particularly for complex interventions), 68,69 and iterating based on periodic feedback.”
Tricco, A. C., Hezam, A., Parker, A., Nincic, V., Harris, C., Fennelly, O., Thomas, S. M., Ghassemi, M., McGowan, J., Paprica, P. A., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-065845 · Cited by 32
“23 For patient, clinician and system-level outcomes, a pre-existing framework 24 25 was also used by one reviewer (VN).”
Self-citationMlodzinski, E., Wardi, G., Viglione, C., Nemati, S., Crotty Alexander, L., & Malhotra, A. (2023) · JMIR Perioperative Medicine · doi:10.2196/41056 · Cited by 28
Metadata
- Authors
- Verma, A. A., Murray, J., Greiner, R., Cohen, J. P., Shojania, K. G., Ghassemi, M., Straus, S. E., Pou-Prom, C., & Mamdani, M.
- Venue
- Canadian Medical Association Journal
- Publisher
- CMA Joule Inc.
- Published
- 2021-08-29
- Volume
- 193
- Issue
- 34
- Pages
- E1351-E1357
- ISSN
- 0820-3946 (ISSN-L)
- 1488-2329
- OpenAlex
- W3197078525
- Cited by
- 136
- References
- 21
- FWCI
- 5.33
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.