The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness
Open accessAbstract
Machine learning, artificial intelligence, and other modern statistical methods are providing new opportunities to operationalise previously untapped and rapidly growing sources of data for patient benefit. Despite much promising research currently being undertaken, particularly in imaging, the literature as a whole lacks transparency, clear reporting to facilitate replicability, exploration for potential ethical concerns, and clear demonstrations of effectiveness. Among the many reasons why these problems exist, one of the most important (for which we provide a preliminary solution here) is the current lack of best practice guidance specific to machine learning and artificial intelligence. However, we believe that interdisciplinary groups pursuing research and impact projects involving machine learning and artificial intelligence for health would benefit from explicitly addressing a series of questions concerning transparency, reproducibility, ethics, and effectiveness (TREE). The 20 critical questions proposed here provide a framework for research groups to inform the design, conduct, and reporting; for editors and peer reviewers to evaluate contributions to the literature; and for patients, clinicians and policy makers to critically appraise where new findings may deliver patient benefit.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Benchimol, E. I., Smeeth, L., Guttmann, A., Harron, K., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E., Langan, S. M., & RECORD Working Committee (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001885 · Cited by 5,749
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · cited 1×
“40 We would urge researchers to keep this in mind when carrying out such comparisons.”
Background
Cited by
Revolutionizing healthcare: the role of artificial intelligence in clinical practice
Alowais, S. A., Alghamdi, S. S., Alsuhebany, N., Alqahtani, T., Alshaya, A. I., Almohareb, S. N., Aldairem, A., Alrashed, M., Bin Saleh, K., Badreldin, H. A., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04698-z · Cited by 3,146
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Self-citationDiagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Artificial intelligence-enhanced electrocardiography in cardiovascular disease management
Siontis, K. C., Noseworthy, P. A., Attia, Z. I., & Friedman, P. A. (2021) · Nature Reviews Cardiology · doi:10.1038/s41569-020-00503-2 · Cited by 914
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
Self-citationVasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665
Self-citationLekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“9 To have a real impact at scale, such guidelines for responsible and trustworthy AI must be obtained through wide consensus involving international and interdisciplinary experts.”
Self-citationResponsible Artificial Intelligence Governance: A Review and Research Framework
Papagiannidis, E., Mikalef, P., & Conboy, K. (2025) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2024.101885 · Cited by 565
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
Self-citationArtificial intelligence in dental research: Checklist for authors, reviewers, readers
Schwendicke, F., Singh, T., Lee, J.-H., Gaudin, R., Chaurasia, A., Wiegand, T., Uribe, S., & Krois, J. (2021) · Journal of Dentistry · doi:10.1016/j.jdent.2021.103610 · Cited by 360
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358
Machine learning based early warning system enables accurate mortality risk prediction for COVID-19
Gao, Y., Cai, G.-Y., Fang, W., Li, H.-Y., Wang, S.-Y., Chen, L., Yu, Y., Liu, D., Xu, S., Cui, P.-F., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18684-2 · Cited by 349
Designing deep learning studies in cancer diagnostics
Kleppe, A., Skrede, O.-J., De Raedt, S., Liestøl, K., Kerr, D. J., & Danielsen, H. E. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-020-00327-9 · Cited by 316
The value of standards for health datasets in artificial intelligence-based applications
Arora, A., Alderman, J. E., Palmer, J., Ganapathi, S., Laws, E., McCradden, M. D., Oakden-Rayner, L., Pfohl, S. R., Ghassemi, M., McKay, F., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02608-w · Cited by 294
Self-citationThe need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 276
Chin, M. H., Afsar-Manesh, N., Bierman, A. S., Chang, C., Colón-Rodríguez, C. J., Dullabh, P., Duran, D. G., Fair, M., Hernandez-Boussard, T., Hightower, M., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.45050 · Cited by 275
Deep learning and the electrocardiogram: review of the current state-of-the-art
Somani, S., Russak, A. J., Richter, F., Zhao, S., Vaid, A., Chaudhry, F., De Freitas, J. K., Naik, N., Miotto, R., Nadkarni, G. N., et al. (2021) · EP Europace · doi:10.1093/europace/euaa377 · Cited by 264
Cowley, L. E., Farewell, D. M., Maguire, S., & Kemp, A. M. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 263
Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review
Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y., & Kann, B. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.33946 · Cited by 236
Time to reality check the promises of machine learning-powered precision medicine.
Wilkinson, J., Arnold, K. F., Murray, E. J., van Smeden, M., Carr, K., Sippy, R., de Kamps, M., Beam, A., Konigorski, S., Lippert, C., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30200-4 · Cited by 233
“8 Although news media is filled with enthusiastic stories about novel mac hine learning applications, 9–12 a systematic review com paring the performance of deep learning versus health professional assessment in diagnosis of various diseases from medical images makes for sobering reading.”
BackgroundBenefits and Risks of AI in Health Care: Narrative Review
Chustecki, M. (2024) · Interactive Journal of Medical Research · doi:10.2196/53616 · Cited by 208
“Smartphone apps that use ML algorithms can help patients with chronic diseases better manage their conditions, leading to healthier populations and reduced health care expenses [36].”
A time-resolved proteomic and prognostic map of COVID-19
Demichev, V., Tober-Lau, P., Lemke, O., Nazarenko, T., Thibeault, C., Whitwell, H., Röhl, A., Freiwald, A., Szyrwiel, L., Ludwig, D., et al. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.005 · Cited by 200
Machine learning in precision diabetes care and cardiovascular risk prediction
Oikonomou, E. K., & Khera, R. (2023) · Cardiovascular Diabetology · doi:10.1186/s12933-023-01985-3 · Cited by 190
Babel, A., Taneja, R., Mondello Malvestiti, F., Monaco, A., & Donde, S. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.669869 · Cited by 199
A Clinician's Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies
Faes, L., Liu, X., Wagner, S. K., Fu, D. J., Balaskas, K., Sim, D. A., Bachmann, L. M., Keane, P. A., & Denniston, A. K. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.7 · Cited by 162
Wubineh, B. Z., Deriba, F. G., & Woldeyohannis, M. M. (2024) · Urologic Oncology: Seminars and Original Investigations · doi:10.1016/j.urolonc.2023.11.019 · Cited by 141
Zhou, Q., Chen, Z.-H., Cao, Y.-H., & Peng, S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00524-2 · Cited by 126
Cai, Y., Cai, Y.-Q., Tang, L.-Y., Wang, Y.-H., Gong, M., Jing, T.-C., Li, H.-J., Li-Ling, J., Hu, W., Yin, Z., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03273-7 · Cited by 122
Andaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121
Self-citationHurkmans, C., Bibault, J.-E., Brock, K. K., van Elmpt, W., Feng, M., David Fuller, C., Jereczek-Fossa, B. A., Korreman, S., Landry, G., Madesta, F., et al. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110345 · Cited by 110
Wells, B. J., Nguyen, H. M., McWilliams, A., Pallini, M., Bovi, A., Kuzma, A., Kramer, J., Chou, S.-H., Hetherington, T., Corn, P., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01900-y · Cited by 90
Garcia Santa Cruz, B., Bossa, M. N., Sölter, J., & Husch, A. D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102225 · Cited by 89
Mentis, A.-F. A., Lee, D., & Roussos, P. (2024) · Molecular Psychiatry · doi:10.1038/s41380-023-02047-6 · Cited by 75
The Debate on the Ethics of AI in Health Care: a Reconstruction and Critical Review
Morley, J., Machado, C., Burr, C., Cowls, J., Taddeo, M., & Floridi, L. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3486518 · Cited by 66
Artificial intelligence bias in medical system designs: a systematic review
Kumar, A., Aelgani, V., Vohra, R., Gupta, S. K., Bhagawati, M., Paul, S., Saba, L., Suri, N., Khanna, N. N., Laird, J. R., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16029-x · Cited by 65
Wang, Z., Myles, P., & Tucker, A. (2021) · Computational Intelligence · doi:10.1111/coin.12427 · Cited by 65
Self-citationMust-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning
Koçak, B., Cuocolo, R., Santos, D. P. D., Stanzione, A., & Ugga, L. (2023) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2022.2022-11-51 · Cited by 64
Artificial Intelligence: Exploring the Future of Innovation in Allergy Immunology
MacMath, D., Chen, M., & Khoury, P. (2023) · Current Allergy and Asthma Reports · doi:10.1007/s11882-023-01084-z · Cited by 62
Improving the quality of machine learning in health applications and clinical research
Mateen, B. A., Liley, J., Denniston, A. K., Holmes, C. C., & Vollmer, S. J. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00239-1 · Cited by 61
Self-citationJauk, S., Kramer, D., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01727-6 · Cited by 59
Applications of Artificial Intelligence in Dental Medicine: A Critical Review
Sitaras, S., Tsolakis, I. A., Gelsini, M., Tsolakis, A. I., Schwendicke, F., Wolf, T. G., & Perlea, P. (2025) · International Dental Journal · doi:10.1016/j.identj.2024.11.009 · Cited by 42
Predicting bloodstream infection outcome using machine learning
Zoabi, Y., Kehat, O., Lahav, D., Weiss-Meilik, A., Adler, A., & Shomron, N. (2021) · Scientific Reports · doi:10.1038/s41598-021-99105-2 · Cited by 37
Coombs, L., Orlando, A., Wang, X., Shaw, P., Rich, A. S., Lakhtakia, S., Titchener, K., Adamson, B., Miksad, R. A., & Mooney, K. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00660-3 · Cited by 27
Assessing the net benefit of machine learning models in the presence of resource constraints
Singh, K., Shah, N. H., & Vickers, A. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad006 · Cited by 21
Decoding machine learning in nursing research: A scoping review of effective algorithms
Choi, J., Lee, H., & Kim‐Godwin, Y. (2025) · Journal of Nursing Scholarship · doi:10.1111/jnu.13026 · Cited by 15
Moorthy, U. M. K., Muthukumaran, A. M. J., Kaliyaperumal, V., Jayakumar, S., & Vijayaraghavan, K. A. (2025) · Explainable Artificial Intelligence in the Healthcare Industry · doi:10.1002/9781394249312.ch23 · Cited by 8
Exploring the Applications of Artificial Intelligence in Marketing: A Topic Modelling Analysis
NOVA Information Management School (NOVA IMS), L., Rita, P., Omran, W., University of Minho, F. O. E. A. M., Ramos, R. F., Instituto Politécnico de Coimbra, E., Costa, T., & NOVA Information Management School (NOVA IMS), L. (2025) · Tourism & Management Studies · doi:10.18089/tms.20250103 · Cited by 7
Metadata
- Authors
- Vollmer, S., Mateen, B. A., Bohner, G., Király, F. J., Ghani, R., Jonsson, P., Cumbers, S., Jonas, A., McAllister, K. S. L., Myles, P., Grainger, D., Birse, M., Branson, R., Moons, K. G. M., Collins, G. S., Ioannidis, J. P. A., Holmes, C., & Hemingway, H.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2020-03-20
- Volume
- 368
- Pages
- l6927
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W3011403448
- Cited by
- 497
- References
- 67
- FWCI
- 15.91
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.