Cited by 188Artificial Intelligence in Healthcare and EducationPrivacy-Preserving Technologies in DataMachine Learning in Healthcare

Federated benchmarking of medical artificial intelligence with MedPerf

Open access
Karargyris, Alexandros, Umeton, Renato, Sheller, Micah J., Aristizabal, Alejandro, George, Johnu, Wuest, Anna, Pati, Sarthak, Kassem, Hasan +69 more·2023·Nature Machine Intelligence

Abstract

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-023-00652-2

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. Crimi, A. & Bakas, S.) 3-24 (Springer, 2022).
  2. Martel, A. L. et al.) 343-352 (Springer, 2020).
  3. Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraATS challenge. Preprint at https://arxiv.org/abs/1811.02629 (2018). · arXiv
  4. Liu, Q. & Schlangen, D.) 38-45 (Association for Computational Linguistics, 2020).
  5. Ul Haq, H., Kocaman, V. & Talby, D. Deeper clinical document understanding using relation extraction. In Proc. Workshop on Scientific Document Understanding (eds Veyseh, A. P. B. et al.) Vol. 3164 (CEUR-WS, 2022).
  6. Ul Haq, H., Kocaman, V. & Talby, D. in Multimodal AI in Healthcare: A Paradigm Shift in Health Intelligence (eds Shaban-Nejad, A. et al.) 361-375 (Springer, 2022).
  7. SIG for Challenges (MICCAI, 2023); http://www.miccai.org/ special-interest-groups/challenges/
  8. Frangi, A. F. et al.) 388-395 (Springer, 2018).
  9. Harvard T.H. Chan School of Public Health, Boston, MA, USA. 6 Massachusetts Institute of Technology, Cambridge, MA, USA.
  10. Broad Institute of MIT and Harvard, Cambridge, MA, USA. 49 These authors contributed equally: Alexandros Karargyris, Renato Umeton, Micah J. Sheller. 50 These authors jointly supervised this work: Spyridon Bakas, Peter Mattson. *A list of authors and their affiliations appears at the end of the paper. e-mail: alex@mlcommons.org; renato_umeton@dfci.harvard.edu; micah.j.sheller@intel.com; jason_johnson@dfci.harvard.edu; sbakas@upenn.edu; peter@mlcommons.org

Cited by

Sorted by citations

Authors
Karargyris, A., Umeton, R., Sheller, M. J., Aristizabal, A., George, J., Wuest, A., Pati, S., Kassem, H., Zenk, M., Baid, U., Narayana Moorthy, P., Chowdhury, A., Guo, J., Nalawade, S., Rosenthal, J., Kanter, D., Xenochristou, M., Beutel, D. J., Chung, V., Bergquist, T., Eddy, J., Abid, A., Tunstall, L., Sanseviero, O., Dimitriadis, D., Qian, Y., Xu, X., Liu, Y., Goh, R. S. M., Bala, S., Bittorf, V., Puchala, S. R., Ricciuti, B., Samineni, S., Sengupta, E., Chaudhari, A., Coleman, C., Desinghu, B., Diamos, G., Dutta, D., Feddema, D., Fursin, G., Huang, X., Kashyap, S., Lane, N., Mallick, I., FeTS Consortium, BraTS-2020 Consortium, AI4SafeChole Consortium, Mascagni, P., Mehta, V., Moraes, C. F., Natarajan, V., Nikolov, N., Padoy, N., Pekhimenko, G., Reddi, V. J., Reina, G. A., Ribalta, P., Singh, A., Thiagarajan, J. J., Albrecht, J., Wolf, T., Miller, G., Fu, H., Shah, P., Xu, D., Yadav, P., Talby, D., Awad, M. M., Howard, J. P., Rosenthal, M., Marchionni, L., Loda, M., Johnson, J. M., Bakas, S., & Mattson, P.
Affiliations
  • Institut de Chirurgie Guidée par l'Image (FR)
  • Harvard University (US)
  • Intel (United States) (US)
  • Nutanix (United States) (US)
  • University of Pennsylvania (US)
  • Université de Strasbourg (FR)
  • German Cancer Research Center (DE)
  • Dana-Farber Cancer Institute (US)
  • Stanford University (US)
  • University of Cambridge (GB)
  • Sage Bionetworks (US)
  • Hugging Face (US)
  • Microsoft (United States) (US)
  • Agency for Science, Technology and Research (SG)
  • Supermicro (United States) (US)
  • Meta (United States) (US)
  • Stanford Medicine (US)
  • Rutgers, The State University of New Jersey (US)
  • Red Hat (United States) (US)
  • Cisco Systems (United States) (US)
  • IBM (United States) (US)
  • Tata Medical Center (IN)
  • Agostino Gemelli University Polyclinic (IT)
  • University of Trento (IT)
  • Google (United States) (US)
  • University of Toronto (CA)
  • Nvidia (United States) (US)
  • Massachusetts Institute of Technology (US)
  • Lawrence Livermore National Laboratory (US)
  • University of York (GB)
  • John Snow (United States) (US)
  • The University of Queensland (AU)
  • Brigham and Women's Hospital (US)
  • Weill Cornell Medicine (US)
  • Broad Institute (US)
Corresponding author
  • Karargyris, Alexandros
  • Umeton, Renato
  • Sheller, Micah J.
  • Johnson, Jason M.
  • Bakas, Spyridon
  • Mattson, Peter
Venue
Nature Machine Intelligence
Published
2023-07-17
Volume
5
Issue
7
Pages
799-810
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4384561507
Cited by
188
References
51
Citations by year
2026: 74 · 2025: 55 · 2024: 49 · 2023: 9 · 2022: 1
FWCI
7.47
Type
journal article
Language
en
Open access
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Research Foundation (AISG2-TC-)
  • National Research Foundation Singapore (AISG2-TC-2021-003)
  • Agence Nationale de la Recherche (ANR-10-IAHU-0002, ANR-10–IAHU-02, ANR-20-CHIA-0029, ANR-10)
  • National Institutes of Health (16,17)
  • National Cancer Institute (U24CA248265, U24CA189523, U01CA242871)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
artificial intelligence in healthcare, healthcare data privacy, model evaluation, cloud deployment, on-premise deployment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review