Cited by 242Privacy-Preserving Technologies in DataArtificial Intelligence in Healthcare and EducationDomain Adaptation and Few-Shot Learning

Federated learning improves site performance in multicenter deep learning without data sharing

Open access
Sarma, Karthik V, Harmon, Stephanie, Sanford, Thomas, Roth, Holger R, Xu, Ziyue, Tetreault, Jesse, Xu, Daguang, Flores, Mona G +11 more·2021·Journal of the American Medical Informatics Association

Abstract

Objective: To demonstrate enabling multi-institutional training without centralizing or sharing the underlying physical data via federated learning (FL). Materials and Methods: Deep learning models were trained at each participating institution using local clinical data, and an additional model was trained using FL across all of the institutions. Results: We found that the FL model exhibited superior performance and generalizability to the models trained at single institutions, with an overall performance level that was significantly better than that of any of the institutional models alone when evaluated on held-out test sets from each institution and an outside challenge dataset. Discussion: The power of FL was successfully demonstrated across 3 academic institutions while avoiding the privacy risk associated with the transfer and pooling of patient data. Conclusion: Federated learning is an effective methodology that merits further study to enable accelerated development of models across institutions, enabling greater generalizability in clinical use.

AI snapshot

Written from the paper’s full text.

TL;DR

In a 3-institution study using real prostate MRI, federated learning produced a single model without sharing patient data. The FL model outperformed each institution’s private model on held-out institutional test sets and an external challenge dataset.

Key findings

  • The FL model achieved an overall private-test-set mean Dice of 0.895 ± 0.036, significantly better than each private model (P < .001); private model overall Dice values were 0.833 ± 0.131 (NCI), 0.854 ± 0.074 (SUNY), and 0.745 ± 0.178 (UCLA).
  • On the external PROSTATEx challenge dataset (n = 343), the FL model achieved 0.889 ± 0.036, significantly better than the private models: NCI 0.872 ± 0.062, SUNY 0.838 ± 0.043, UCLA 0.812 ± 0.136 (P < .001 for all comparisons).
  • Private models performed well on their own institutional test sets (Dice range 0.883–0.925) but had diminished performance on other institutions’ test sets (Dice range 0.575–0.887).

Methods

  • Retrospective multicenter study at UCLA, SUNY Upstate, and NCI: each institution collected 1 prostate MRI from 100 patients, with axial T2W images at 3T and expert whole-prostate segmentations; each private dataset was split into 80 training and 20 held-out test images. An external challenge dataset of 343 axial T2W prostate images from SPIE-AAPM-NCI PROSTATEx was also used.
  • The model was a 3D Anisotropic Hybrid Network trained with soft Dice loss and Adam; each private training set was split into 5 sets of 16 images to train 5 submodels combined into an ensemble. All models were trained for 300 epochs.
  • Federated learning used a cloud-based aggregation server; after each epoch, model weights and validation metrics were sent from each institution, combined by a weighted average based on validation mean Dice, and redistributed. Evaluation used Dice coefficient and 2-sided paired t-tests comparing private models to the FL model on the combined private test set and the challenge dataset.

Limitations

  • Stated: the authors did not address the potential for an inside actor to recover underlying patient data through a model inversion attack on shared weights; they suggest calibrated distortion to shared weights as a possible future enhancement.
  • Stated: the task (prostate segmentation on T2-weighted MRI) is relatively simple, and all private models achieved high performance on their own institutional datasets, so the study could not demonstrate that an FL-trained model would significantly outperform a single-site-trained model on that single site’s data.
  • Stated: because private datasets were similarly sized, the authors did not explore varying the federated aggregation methodology; adding more institutions may introduce challenges in imaging data quality heterogeneity, governance, intellectual property, and model generalizability, possibly requiring private fine-tuning. Evident from the text: only 3 academic institutions and one segmentation task were tested, and no formal privacy attack evaluation was performed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa341

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 9 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Works that examine the practical application of FL in radiological applications are still limited.

    “However, works that examine the practical application of FL in radiological applications are still limited.20,21”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [20] Roth

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [21] Sheller, 2020, Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data, Sci Rep source

      The Methods > Data and Discussion passages describe this paper’s evaluation of federated learning on radiographic brain imaging, so it is an example of the kind of work the citing sentence identifies. They do not establish that such works are limited in number. No page numbers are available in our copy. (LLM, conf=0.81)

      Experimental — accuracy not yet measured
  • Findingown data

    The private models exhibited varied performance on the challenge dataset (Dice 0.812-0.872).

    “The private models exhibited varied performance on the challenge dataset (Dice coefficient range: 0.812–0.872).”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    The FL model did not significantly outperform a single-site-trained model on that single site's data.

    “we were unable to demonstrate the expected benefit that an FL-trained model would significantly outperform a single-site-trained model on that single site’s data.”
    Quote in the paperSubstantial claim
  • Study design and conductreasoning

    No private data were transferred or shared across institutions during the study.

    “At no point during this study were private data transferred or shared across institutions.”
    • Also stated in: § DISCUSSION · plain
      “Since no transfer of protected health information (or even deidentified health information) was required, we were able to address the privacy and data governance limitations inherent to multicenter studies through the use of simplified 2-way collaboration agreements, rather than requiring the negotiation of a complex 4-way collaboration and material transfer agreement that would have been required”
    Quote in the paperSubstantial claim
  • Interpretationown data

    The power of federated learning was successfully demonstrated across 3 academic institutions using real clinical prostate imaging data.

    “The power of federated learning was successfully demonstrated across 3 academic institutions using real clinical prostate imaging data.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 6
  • May combine two claims. 8
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Previous work has demonstrated the potential utility of FL for model training, generally using publicly available data to simulate multi-institutional training.

    “Previous work has demonstrated the potential utility of FL for model training, generally using publicly available data to simulate multi-institutional training.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [20] Roth

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [21] Sheller, 2020, Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data, Sci Rep source
  • May combine two claims.

    Premisecitation

    Deep learning models in medical imaging are often trained on single-institution datasets and often suffer from poor performance when transferred to other institutions with differing protocols, equipment, or patient populations.

    “Such models often suffer from poor performance when transferred to other institutions with differing protocols, equipment, or patient populations.14,15”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [14] Yasaka, 2018, Deep learning and artificial intelligence in radiology: current applications and future directions, PLOS Med source
    • cites [15] De Fauw, 2018, Clinically applicable deep learning for diagnosis and referral in retinal disease, Nat Med source

      The Introduction (p. 1) identifies generalization to new populations and devices as a challenge. In “Generalization to a New Scanning Device Type” (p. 7), the authors report a 46.6% referral-error rate when their model was applied to scans from a different device without retraining. That supports the equipment-transfer point, but the study does not establish that models often perform poorly across institutions or that differing protocols or patient populations caused such declines. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • May combine two claims.

    Findingown data

    The private models performed well on their own private test sets (Dice 0.883-0.925) but had diminished performance on other private test sets (Dice 0.575-0.887), with overall mean Dice coefficients between 0.745 and 0.833.

    “The private models performed well on their own private test sets (Dice coefficient range: 0.883–0.925) but had diminished performance on the other private test sets (Dice coefficient range: 0.575–0.887). This led to overall mean Dice coefficients between 0.745 and 0.833 for the private models.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    On the external ProstateX challenge dataset (n=343), the FL model achieved an overall mean Dice coefficient of 0.889 and was significantly superior to each private model (P < .001).

    “The generic FL model outperformed each of the private models, with an overall mean Dice coefficient of 0.889. The statistical analysis again demonstrated that the FL model was significantly superior to any of the private models (P < .001).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    On the combined private test sets (n=60), the FL model achieved an overall mean Dice coefficient of 0.895 (range 0.880-0.920) and was significantly superior to each private model (P < .001 for all comparisons).

    “The FL model exhibited private test set mean Dice coefficients between 0.880 and 0.920, yielding an overall result of 0.895. The statistical analysis using 2-sided paired t-tests demonstrated that the FL model was significantly superior to any of the private models (P < .001 for all comparisons).”
    • Also stated in: § DISCUSSION · plain
      “The FL model that we trained performed well across all of the private datasets, yielding an overall performance level that was significantly better than that of any of the private models alone.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    The federated model demonstrated improved performance across both held-out test sets from each institution and an external test set, validating the FL paradigm.

    “The federated model demonstrated improved performance across both held-out test sets from each institution and an external test set, validating the FL paradigm.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    In the worst case, the method is no less secure than current practice because only model weights are ever transmitted.

    “in the worst case, our method is no less secure as only model weights are ever transmitted.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All 3 academic institutions had IRB approval for review and image analysis, with written informed patient consent or waiver of patient consent.

    “All 3 academic institutions had IRB approval for review and image analysis, with written informed patient consent or waiver of patient consent.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study did not attempt to address the potential for an inside actor to recover underlying patient data through a model inversion attack on the trained weights shared during federated learning.

    “we did not attempt to address the potential for an inside actor (ie, 1 of the participating institutions) to attempt to recover the underlying patient data through a model inversion attack on the trained weights shared during federated learning.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The sharing of trained model weights is an accepted practice within healthcare.

    “the sharing of trained model weights is an accepted practice within healthcare,28,29”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [28] Soni

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [29] Liu, 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Federated learning can be reduced to practice using real-world private clinical data across multiple institutions, creating a model with improved generalizability both within participating institutions and with outside data.

    “Our work shows that FL can be reduced to practice using real-world private clinical data across multiple institutions, and that this approach creates a model that demonstrates improved generalizability both within the participating institutions and with outside data.”
    • Also stated in: § Results · plain
      “We found that the FL model exhibited superior performance and generalizability to the models trained at single institutions, with an overall performance level that was significantly better than that of any of the institutional models alone when evaluated on held-out test sets from each institution and an outside challenge dataset.”
    • Also stated in: § CONCLUSION · boosted
      “The federated model demonstrated improved performance across both held-out test sets from each institution and an external test set, validating the FL paradigm.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Model inversion is a technical risk that cannot be ruled out.

    “though model inversion is a technical risk that cannot be ruled out”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    A limitation of FL is that no single model ever sees a complete picture of the universe of potential inputs during training, placing pressure on the federated aggregation function to adequately distribute knowledge from each site into the model.

    “However, a limitation of this approach is that no single model ever “sees” a complete picture of the universe of potential inputs during the training phase, thus placing pressure on the federated aggregation function to adequately distribute knowledge from each site into the model.”
    Quote in the paperNot a substantial claim
13 of 13
  • The paper relies on this without support.

    The practical risk of model inversion by an inside actor outside crafted malintent is low due to the weight averaging scheme.

    “we empirically believe that the practical risk of inversion outside of crafted malintent on the part of study designers to be low due to the weight averaging scheme in place.”
  • The paper relies on this without support.

    Medical imaging data in most cases is siloed within provider institutions, and assembling large-scale datasets traditionally requires transfer of data between silos.

    “However, medical imaging data in most cases is siloed within provider institutions, and, as a result, assembling large-scale datasets traditionally requires the transfer of data between these silos.”
  • The paper relies on this without support.

    Very few public large-scale pooled medical image datasets exist.

    “As a result, very few public large-scale pooled medical image datasets exist.”
  • The paper relies on this without support.

    Deep learning models are primarily interpolators, not extrapolators—they perform best when presented with inputs similar to training data.

    “This large volume is required because deep learning models are primarily interpolators, not extrapolators—that is, they perform best when presented with inputs that are similar to the data that those models were trained on.”
  • The paper relies on this without support.

    The task of prostate segmentation on T2-weighted MRI is relatively simple.

    “the task we used (prostate segmentation on T2-weighted MRI) is relatively simple”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Federated learning provides an alternative paradigm for accurate and generalizable machine learning by only sharing numerical model updates from decentralized data.

    “Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Private-model overall range omits the SUNY result

    Minor

    The stated upper bound is 0.833, but Table 3 reports an overall mean Dice coefficient of 0.854 for the SUNY private model.

1 of 1

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Sarma, K. V., Harmon, S., Sanford, T., Roth, H. R., Xu, Z., Tetreault, J., Xu, D., Flores, M. G., Raman, A. G., Kulkarni, R., Wood, B. J., Choyke, P. L., Priester, A. M., Marks, L. S., Raman, S. S., Enzmann, D., Turkbey, B., Speier, W., & Arnold, C. W.
Affiliations
  • University of California, Los Angeles (US)
  • National Institutes of Health (US)
  • SUNY Upstate Medical University (US)
  • Nvidia (United States) (US)
  • National Cancer Institute, National Institutes of Health, Bethesda, Maryland, USA
Corresponding author
Arnold, Corey W
Venue
Journal of the American Medical Informatics Association
Published
2021-02-04
Volume
28
Issue
6
Pages
1259-1264
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W3128304793
Cited by
242
References
25
Citations by year
2026: 27 · 2025: 51 · 2024: 54 · 2023: 48 · 2022: 34 · 2021: 18
FWCI
9.90
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • NIH NCI (F30CA210329, R21CA220352, P50CA092131, ZIDBC011242, Z1ACL040015, HHSN261200800001E)
  • NIH NIGMS (GM08042)
  • NIH
  • AMA Foundation
  • NVIDIA Corporation Academic Hardware Grant
  • NIH Center for Interventional Oncology
  • Intramural Research Program of the NIH, and a cooperative research and development agreement (CRADA) between NIH and nVIDIA
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
federated learning, data privacy, data sharing, model generalization, clinical data
MeSH terms
Deep Learning, Humans, Privacy, Information Dissemination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review