The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Federated benchmarking of medical artificial intelligence with MedPerf
Open accessAbstract
Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Beyond training, AI models must also be tested across multiple independent sites and evaluated for overall accuracy and potential bias (162).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [162]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Self-citationk-ANONYMITY: A MODEL FOR PROTECTING PRIVACY
SWEENEY, L. (2002) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488502001648 · Cited by 8,571
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 2×
“…studies have shown that models trained with data from limited and specific clinical settings are often biased with respect to specific patient populations 11–13 ; such data biases can lead to models that seem promising during development but have lower performance in wider deployment 14,15 .”
The future of digital health with federated learning
Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 4,015 · cited 1×
“In a federated evaluation platform, data are always accessed and analysed locally.”
Self-citationThe EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“…studies have shown that models trained with data from limited and specific clinical settings are often biased with respect to specific patient populations 11–13 ; such data biases can lead to models that seem promising during development but have lower performance in wider deployment 14,15 .”
Sheller, M. J., Edwards, B., Reina, G. A., Martin, J., Pati, S., Kotrotsou, A., Milchenko, M., Xu, W., Marcus, D., Colen, R. R., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-69250-1 · Cited by 1,646
Self-citationDeep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · cited 1×
“• Data model standardization efforts to enable collaboration between institutions, such as the OMOP Common Data Model 80,81 , possibly leveraging the highly multimodal nature of biomedical data 82 .”
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,107 · cited 2×
“• Data model standardization efforts to enable collaboration between institutions, such as the OMOP Common Data Model 80,81 , possibly leveraging the highly multimodal nature of biomedical data 82 .”
Federated learning for predicting clinical outcomes in patients with COVID-19
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01506-3 · Cited by 842
Self-citationBeede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614
Sheller, M. J., Reina, G. A., Edwards, B., Martin, J., & Bakas, S. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11723-8_9 · Cited by 609
Self-citationEnd-to-end privacy preserving deep learning on multi-institutional medical imaging
Kaissis, G., Ziller, A., Passerat-Palmbach, J., Ryffel, T., Usynin, D., Trask, A., Lima, I., Mancuso, J., Jungmann, F., Steinborn, M.-M., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00337-8 · Cited by 513
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494
The “inconvenient truth” about AI in healthcare
Panch, T., Mattie, H., & Celi, L. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0155-4 · Cited by 473
Why rankings of biomedical image analysis competitions should be interpreted with care
Maier‐Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Godau, P., Arbel, T., Bogunović, H., Bradley, A. P., Carass, A., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07619-7 · Cited by 428
Yao, X., Rushlow, D. R., Inselman, J. W., McCoy, R. G., Thacher, T. D., Behnken, E. M., Bernard, M. E., Rosas, S. L., Akfaly, A., Misra, A., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01335-4 · Cited by 424
Artificial intelligence for clinical oncology
Kann, B. H., Hosny, A., & Aerts, H. J. W. L. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.04.002 · Cited by 404
HIPAA regulations - a new era of medical-record privacy?
Annas, G. J. (2003) · New England Journal of Medicine · doi:10.1056/nejmlim035027 · Cited by 402
Federated learning enables big data for rare cancer boundary detection
Pati, S., Baid, U., Edwards, B., Sheller, M., Wang, S.-H., Reina, G. A., Foley, P., Gruzdev, A., Karkada, D., Davatzikos, C., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33407-5 · Cited by 394
Self-citationWinkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1735 · Cited by 365
Causality matters in medical imaging
Castro, D. C., Walker, I., & Glocker, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17478-w · Cited by 364
A deep learning algorithm to predict risk of pancreatic cancer from disease trajectories
Placido, D., Yuan, B., Hjaltelin, J. X., Zheng, C., Haue, A. D., Chmura, P. J., Yuan, C., Kim, J., Umeton, R., Antell, G., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02332-5 · Cited by 349
Self-citationDatasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · cited 1×
“Benefits of using the Hugging Face Hub include the fact that artifacts can be accessed from Hugging Face’s popular open-source libraries, such as datasets 47 , transformers 48 and evaluation 49 .”
Self-citationPantanowitz, L., Sharma, A., Carter, A. B., Kurc, T., Sussman, A., & Saltz, J. (2018) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_69_18 · Cited by 282
From knowledge to action: the impact of benchmarking on organizational performance
Drew, S. (1997) · Long Range Planning · doi:10.1016/s0024-6301(97)90262-4 · Cited by 261
Using HL7 FHIR to achieve interoperability in patient health record
Saripalle, R. K., Runyan, C., & Mitchell, R. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103188 · Cited by 259
TeCNO: Surgical Phase Recognition with Multi-stage Temporal Convolutional Networks
Czempiel, T., Paschali, M., Keicher, M., Simson, W. A., Feußner, H., Kim, S. T., & Navab, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59716-0_33 · Cited by 257
OpenFL: the open federated learning library
Foley, P., Sheller, M., Edwards, B., Pati, S., Riviera, W., Sharma, M., Moorthy, P. N., Wang, S.-H., Martin, J., Mirhaji, P., et al. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac97d9 · Cited by 253
Self-citationGeographic Distribution of US Cohorts Used to Train Deep Learning Algorithms
Kaushal, A., Altman, R., & Langlotz, C. (2020) · JAMA · doi:10.1001/jama.2020.12067 · Cited by 239
Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review
Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y., & Kann, B. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.33946 · Cited by 236
Ogier du Terrail, J., Leopold, A., Joly, C., Béguier, C., Andreux, M., Maussion, C., Schmauch, B., Tramel, E. W., Bendjebbar, E., Zaslavskiy, M., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02155-w · Cited by 217
Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.01500 · Cited by 206 · cited 2×
“…past efforts such as (1) the development of standardized benchmarking platforms (such as MLPerf, for benchmarking machine learning training 29 and inference across industries in a pre-competitive space 30 ); (2) the implementation of federated learning software libraries such as the Open…”
Self-citationEthics of using and sharing clinical imaging data for artificial intelligence: a proposed framework
Larson, D. B., Magnus, D. C., Lungren, M. P., Shah, N. H., & Langlotz, C. P. (2020) · Radiology · doi:10.1148/radiol.2020192536 · Cited by 199
MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance
Mattson, P., Reddi, V. J., Cheng, C., Coleman, C., Diamos, G., Kanter, D., Micikevicius, P., Patterson, D., Schmuelling, G., Tang, H., et al. (2020) · IEEE Micro · doi:10.1109/mm.2020.2974843 · Cited by 171
Self-citationThe Federated Tumor Segmentation (FeTS) ChallengePreprint
Pati, S., Baid, U., Zenk, M., Edwards, B., Sheller, M., Reina, G. A., Foley, P., Gruzdev, A., Martin, J., Albarqouni, S., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05874 · Cited by 132
Self-citationCommon Limitations of Image Processing Metrics: A Picture Story
Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Arbel, T., Bakas, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.05642 · Cited by 126
Self-citationContinual learning in medical devices: FDA's action plan and beyond
Vokinger, K. N., Feuerriegel, S., & Kesselheim, A. S. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00076-5 · Cited by 95
FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
du Terrail, J. O., Ayed, S.-S., Cyffers, E., Grimberg, F., He, C., Loeb, R., Mangold, P., Marchand, T., Marfoq, O., Mushtaq, E., et al. (2022) · neural information processing systems · doi:10.52202/068431-0384 · Cited by 80
Joint Imaging Platform for Federated Clinical Data Analytics
Scherer, J., Nolden, M., Kleesiek, J., Metzger, J., Kades, K., Schneider, V., Bach, M., Sedlaczek, O., Bucher, A. M., Vogl, T. J., et al. (2020) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00045 · Cited by 76
A Review of Medical Federated Learning: Applications in Oncology and Cancer Research
Chowdhury, A., Kassem, H., Padoy, N., Umeton, R., & Karargyris, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08999-2_1 · Cited by 72
Self-citationPatient data ownership: who owns your health?
Liddell, K., Simon, D. A., & Lucassen, A. (2021) · Journal of Law and the Biosciences · doi:10.1093/jlb/lsab023 · Cited by 69
GaNDLF: the generally nuanced deep learning framework for scalable end-to-end clinical workflows
Pati, S., Thakur, S. P., Hamamcı, İ. E., Baid, U., Baheti, B., Bhalerao, M., Güley, O., Mouchtaris, S., Lang, D., Thermos, S., et al. (2023) · Communications Engineering · doi:10.1038/s44172-023-00066-3 · Cited by 68
Self-citationSpark NLP: Natural Language Understanding at Scale
Kocaman, V., & Talby, D. (2021) · Software Impacts · doi:10.1016/j.simpa.2021.100058 · Cited by 66
Self-citationRosenthal, J., Carelli, R., Omar, M., Brundage, D., Halbert, E., Nyman, J., Hari, S. N., Allen, E. M. V., Marchionni, L., Umeton, R., et al. (2021) · Molecular Cancer Research · doi:10.1158/1541-7786.mcr-21-0665 · Cited by 48
Self-citationAccurate Clinical and Biomedical Named Entity Recognition at Scale
Kocaman, V., & Talby, D. (2022) · Software Impacts · doi:10.1016/j.simpa.2022.100373 · Cited by 45
Self-citationEvaluate & Evaluation on the Hub: Better Best Practices for Data and Model MeasurementsPreprint
von Werra, L., Tunstall, L., Thakur, A., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.01970 · Cited by 38
Self-citationHow to Exploit Weaknesses in Biomedical Challenge Design and Organization
Reinke, A., Eisenmann, M., Onogur, S., Stankovic, M., Godau, P., Full, P. M., Bogunović, H., Landman, B. A., Maier, O., Menze, B. H., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00937-3_45 · Cited by 35
Geleijnse, G., Chiang, R. C.-J., Sieswerda, M. S., Schuurman, M. S., Lee, K. C., van Soest, J., Dekker, A. L. A. J., Lee, W., & Verbeek, X. A. A. M. (2020) · Scientific Reports · doi:10.1038/s41598-020-77476-2 · Cited by 32
Reproducible biomedical benchmarking in the cloud: lessons from crowd-sourced data challenges
Ellrott, K., Buchanan, A., Creason, A., Mason, M. J., Schaffter, T., Hoff, B., Eddy, J. A., Chilton, J. M., Yu, T. V., Stuart, J. M., et al. (2019) · Genome biology · doi:10.1186/s13059-019-1794-0 · Cited by 30
Self-citationSirota-Cohen, C., Rosipko, B., Forsberg, D., & Sunshine, J. L. (2018) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-018-0142-3 · Cited by 21
Mining Adverse Drug Reactions from Unstructured Mediums at Scale
Haq, H. U., Kocaman, V., & Talby, D. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-14771-5_26 · Cited by 20
Self-citationIndustry-Scale Orchestrated Federated Learning for Drug DiscoveryPreprint
Oldenhof, M., Ács, G., Pejó, B., Schuffenhauer, A., Holway, N., Sturm, N., Dieckmann, A., Fortmeier, O., Boniface, E., Mayer, C., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.08871 · Cited by 9
Baid, U., Pati, S., Thakur, S. P., Edwards, B., Sheller, M., Martin, J., & Bakas, S. (2021) · Neuro-Oncology · doi:10.1093/neuonc/noab196.532 · Cited by 7
Self-citationEvaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements
von Werra, L., Tunstall, L., Thakur, A., Luccioni, A. S., Thrush, T., Piktus, A., Marty, F., Rajani, N. F., Mustar, V., & Ngo, H. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.13 · Cited by 7
Self-citation
Research works printed but not linked: 3
- Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraATS challenge. Preprint at https://arxiv.org/abs/1811.02629 (2018). · arXiv
- Ul Haq, H., Kocaman, V. & Talby, D. Deeper clinical document understanding using relation extraction. In Proc. Workshop on Scientific Document Understanding (eds Veyseh, A. P. B. et al.) Vol. 3164 (CEUR-WS, 2022).
- SIG for Challenges (MICCAI, 2023); http://www.miccai.org/special-interest-groups/challenges/
Cited by
Sorted by citations
Privacy preservation for federated learning in health care
Pati, S., Kumar, S., Varma, A., Edwards, B., Lu, C., Qu, L., Wang, J. J., Lakshminarayanan, A., Wang, S.-H., Sheller, M., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.100974 · Cited by 294
Self-citationArtificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions
Lotter, W., Hassett, M. J., Schultz, N., Kehl, K. L., Van Allen, E. M., & Cerami, E. (2024) · Cancer Discovery · doi:10.1158/2159-8290.cd-23-1199 · Cited by 196
Research on Artificial-Intelligence-Assisted Medicine: A Survey on Medical Artificial Intelligence
Gou, F., Liu, J., Xiao, C., & Wu, J. (2024) · Diagnostics · doi:10.3390/diagnostics14141472 · Cited by 129
A comprehensive review on federated learning based models for healthcare applications
Sharma, S., & Guleria, K. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102691 · Cited by 111
Multimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 106
Metadata
- Authors
- Karargyris, A., Umeton, R., Sheller, M. J., Aristizabal, A., George, J., Wuest, A., Pati, S., Kassem, H., Zenk, M., Baid, U., Narayana Moorthy, P., Chowdhury, A., Guo, J., Nalawade, S., Rosenthal, J., Kanter, D., Xenochristou, M., Beutel, D. J., Chung, V., Bergquist, T., Eddy, J., Abid, A., Tunstall, L., Sanseviero, O., Dimitriadis, D., Qian, Y., Xu, X., Liu, Y., Goh, R. S. M., Bala, S., Bittorf, V., Puchala, S. R., Ricciuti, B., Samineni, S., Sengupta, E., Chaudhari, A., Coleman, C., Desinghu, B., Diamos, G., Dutta, D., Feddema, D., Fursin, G., Huang, X., Kashyap, S., Lane, N., Mallick, I., FeTS Consortium, BraTS-2020 Consortium, AI4SafeChole Consortium, Mascagni, P., Mehta, V., Moraes, C. F., Natarajan, V., Nikolov, N., Padoy, N., Pekhimenko, G., Reddi, V. J., Reina, G. A., Ribalta, P., Singh, A., Thiagarajan, J. J., Albrecht, J., Wolf, T., Miller, G., Fu, H., Shah, P., Xu, D., Yadav, P., Talby, D., Awad, M. M., Howard, J. P., Rosenthal, M., Marchionni, L., Loda, M., Johnson, J. M., Bakas, S., & Mattson, P.
- Affiliations
- Institut de Chirurgie Guidée par l'Image (FR)
- Harvard University (US)
- Intel (United States) (US)
- Nutanix (United States) (US)
- University of Pennsylvania (US)
- Université de Strasbourg (FR)
- German Cancer Research Center (DE)
- Dana-Farber Cancer Institute (US)
- Stanford University (US)
- University of Cambridge (GB)
- Sage Bionetworks (US)
- Hugging Face (US)
- Microsoft (United States) (US)
- Agency for Science, Technology and Research (SG)
- Supermicro (United States) (US)
- Meta (United States) (US)
- Stanford Medicine (US)
- Rutgers, The State University of New Jersey (US)
- Red Hat (United States) (US)
- Cisco Systems (United States) (US)
- IBM (United States) (US)
- Tata Medical Center (IN)
- Agostino Gemelli University Polyclinic (IT)
- University of Trento (IT)
- Google (United States) (US)
- University of Toronto (CA)
- Nvidia (United States) (US)
- Massachusetts Institute of Technology (US)
- Lawrence Livermore National Laboratory (US)
- University of York (GB)
- John Snow (United States) (US)
- The University of Queensland (AU)
- Brigham and Women's Hospital (US)
- Weill Cornell Medicine (US)
- Broad Institute (US)
- Corresponding author
- Karargyris, Alexandros
- Umeton, Renato
- Sheller, Micah J.
- Johnson, Jason M.
- Bakas, Spyridon
- Mattson, Peter
- ORCID
- Karargyris, Alexandros
- Umeton, Renato
- Sheller, Micah J.
- Pati, Sarthak
- Kassem, Hasan
- Zenk, Maximilian
- Baid, Ujjwal
- Guo, Junyi
- Nalawade, Sahil
- Rosenthal, Jacob
- Xenochristou, Maria
- Beutel, Daniel J.
- Chung, Verena
- Bergquist, Timothy
- Eddy, James
- Abid, Abubakar
- Dimitriadis, Dimitrios
- Qian, Yiming
- Xu, Xinxing
- Liu, Yong
- Goh, Rick Siow Mong
- Ricciuti, Biagio
- Samineni, Soujanya
- Chaudhari, Akshay
- Fursin, Grigori
- Huang, Xinyuan
- Kashyap, Satyananda
- Lane, Nicholas
- Mallick, Indranil
- Mascagni, Pietro
- Mehta, Virendra
- Moraes, Cassiano Ferro
- Nikolov, Nikola
- Padoy, Nicolas
- Pekhimenko, Gennady
- Reddi, Vijay Janapa
- Reina, G. Anthony
- Singh, Abhishek
- Thiagarajan, Jayaraman J.
- Albrecht, Jacob
- Wolf, Thomas
- Fu, Huazhu
- Shah, Prashant
- Xu, Daguang
- Yadav, Poonam
- Talby, David
- Awad, Mark M.
- Rosenthal, Michael
- Marchionni, Luigi
- Loda, Massimo
- Johnson, Jason M.
- Bakas, Spyridon
- Mattson, Peter
- Venue
- Nature Machine Intelligence
- Published
- 2023-07-17
- Volume
- 5
- Issue
- 7
- Pages
- 799-810
- ISSN
- 2522-5839 (ISSN-L)
- PMC
- PMC11068064
- OpenAlex
- W4384561507
- Cited by
- 188
- References
- 51
- Citations by year
- 2026: 74 · 2025: 55 · 2024: 49 · 2023: 9
- FWCI
- 7.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Research Foundation (AISG2-TC-)
- National Research Foundation Singapore (AISG2-TC-2021-003)
- Agence Nationale de la Recherche (ANR-10-IAHU-0002, ANR-10–IAHU-02, ANR-20-CHIA-0029, ANR-10)
- National Institutes of Health (16,17)
- National Cancer Institute (U24CA248265, U24CA189523, U01CA242871)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- artificial intelligence in healthcare, healthcare data privacy, model evaluation, cloud deployment, on-premise deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
