Reporting and Implementing Interventions Involving Machine Learning and Artificial Intelligence
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Increasingly, interventions aimed at improving care are likely to use such technologies as machine learning and artificial intelligence. However, health care has been relatively late to adopt them. This article provides clinical examples in which machine learning and artificial intelligence are already in use in health care and appear to deliver benefit. Three key bottlenecks toward increasing the pace of diffusion and adoption are methodological issues in evaluation of artificial intelligence-based interventions, reporting standards to enable assessment of model performance, and issues that need to be addressed for an institution to adopt these interventions. Methodological best practices will include external validation, ideally at a different site; use of proactive learning algorithms to correct for site-specific biases and increase robustness as algorithms are deployed across multiple sites; addressing subgroup performance; and communicating to providers the uncertainty of predictions. Regarding reporting, especially important issues are the extent to which implementing standardized approaches for introducing clinical decision support has been followed, describing the data sources, reporting on data assumptions, and addressing biases. Although most health care organizations in the United States have adopted electronic health records, they may be ill prepared to adopt machine learning and artificial intelligence. Several steps can enable this: preparing data, developing tools to get suggestions to clinicians in useful ways, and getting clinicians engaged in the process. Open challenges and the role of regulation in this area are briefly discussed. Although these techniques have enormous potential to improve care and personalize recommendations for individuals, the hype regarding them is tremendous. Organizations will need to approach this domain carefully with knowledgeable partners to obtain the hoped-for benefits and avoid failures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
(2012) · Kidney International Supplements · doi:10.1038/kisup.2012.1 · Cited by 4,326
Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0040-6 · Cited by 1,638
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235
Self-citationCollins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · cited 1×
“One evaluation of how validation was done (23) found many issues: 16% of articles did not include the number of outcomes in the validation set, 54% did not mention the frequency of missing data, and 67% did not report model calibration.”
BackgroundReporting of artificial intelligence prediction models
Collins, G. S., & Moons, K. G. M. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30037-6 · Cited by 701
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598
Self-citationMa, F., Chitta, R., Zhou, J., You, Q., Sun, T., & Gao, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098088 · Cited by 576 · cited 2×
“Multiple investigations have been done of model interpretability for health care applications and CDS, such as RETAIN (37), GBT (38), Dipole (39), and RetainVis (40).”
BackgroundDonzé, J., Aujesky, D., Williams, D., & Schnipper, J. L. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.3023 · Cited by 553 · cited 2×
“For example, did an ML model for readmission get tested against LACE (33) or HOSPITAL (34) scores by using data from the same site? Institutions should be careful to adopt ML systems that have been compared against existing models as a benchmark.”
MethodDesautels, T., Calvert, J., Hoffman, J., Jay, M., Kerem, Y., Shieh, L., Shimabukuro, D., Chettipally, U., Feldman, M. D., Barton, C., et al. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5909 · Cited by 509
Cited by
The potential of artificial intelligence to improve patient safety: a scoping review
Bates, D. W., Levine, D., Syrowatka, A., Kuznetsova, M., Craig, K. J. T., Rui, A., Jackson, G. P., & Rhee, K. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00423-6 · Cited by 323
Self-citationChallenges and opportunities beyond structured data in analysis of electronic health records
Tayefi, M., Ngo, P., Chomutare, T., Dalianis, H., Salvi, E., Budrionis, A., & Godtliebsen, F. (2021) · WIREs Computational Statistics · doi:10.1002/wics.1549 · Cited by 290
Toward a responsible future: recommendations for AI-enabled clinical decision support
Labkoff, S., Oladimeji, B., Kannry, J., Solomonides, A., Leftwich, R., Koski, E., Joseph, A. L., Lopez-Gonzalez, M., Fleisher, L. A., Nolen, K., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae209 · Cited by 252
Artificial intelligence in oncology: path to implementation
Chua, I. S., Gaziel‐Yablowitz, M., Korach, Z. T., Kehl, K. L., Levitan, N. A., Arriaga, Y. E., Jackson, G. P., Bates, D. W., & Hassett, M. (2021) · Cancer Medicine · doi:10.1002/cam4.3935 · Cited by 179
Self-citationArtificial Intelligence in U.S. Health Care Delivery
Sahni, N. R., & Carrus, B. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2204673 · Cited by 175
Metadata
- Authors
- Bates, D. W., Auerbach, A., Schulam, P., Wright, A., & Saria, S.
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2020-06-02
- Volume
- 172
- Issue
- 11_Supplement
- Pages
- S137-S144
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W3031644567
- Cited by
- 114
- References
- 13
- FWCI
- 3.52
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reporting and Implementing Interventions Involving Machine Learning and Artificial Intelligence. https://app.cochat.ai/papers/10.7326/m19-0872
- The paper
- Bates, D. W., Auerbach, A., Schulam, P., Wright, A., & Saria, S. (2020). Reporting and Implementing Interventions Involving Machine Learning and Artificial Intelligence. Annals of Internal Medicine. https://doi.org/10.7326/m19-0872
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.