CoChat
Sign Up
Cited by 394Privacy-Preserving Technologies in DataBrain Tumor Detection and ClassificationArtificial Intelligence in Healthcare and Education

Federated learning enables big data for rare cancer boundary detection

Open access
Pati, Sarthak, Baid, Ujjwal, Edwards, Brandon, Sheller, Micah, Wang, Shih-Han, Reina, G. Anthony, Foley, Patrick, Gruzdev, Alexey +271 more·2022·Nature Communications
DOI 10.1038/s41467-022-33407-5PMC PMC9722782
Open this paper in CoChat — free accountOpen PDF

Abstract

Although machine learning (ML) has shown promise across disciplines, out-of-sample generalizability is concerning. This is currently addressed by sharing multi-site data, but such centralization is challenging/infeasible to scale due to various limitations. Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates. Here we present the largest FL study to-date, involving data from 71 sites across 6 continents, to generate an automatic tumor boundary detector for the rare disease of glioblastoma, reporting the largest such dataset in the literature ( n = 6, 314). We demonstrate a 33% delineation improvement for the surgically targetable tumor, and 23% for the complete tumor extent, over a publicly trained model. We anticipate our study to: 1) enable more healthcare studies informed by large diverse data, ensuring meaningful results for rare diseases and underrepresented populations, 2) facilitate further analyses for glioblastoma by releasing our consensus model, and 3) demonstrate the FL effectiveness at such scale and task-complexity as a paradigm shift for multi-site collaborations, alleviating the need for data-sharing.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper reports the largest real-world federated learning study to date for glioblastoma boundary detection, using data from 6,314 patients across 71 sites on six continents. The final consensus model improved tumor sub-compartment delineation over a publicly trained initial model and showed that federated learning can enable large, diverse rare-disease datasets without sharing patient data.

Key findings

  • The final consensus model improved over the public initial model on collaborators’ local validation data by 27% for enhancing tumor (ET), 33% for tumor core (TC), and 16% for whole tumor (WT); on complete out-of-sample data, improvements were 15% for ET, 27% for TC, and 16% for WT. The abstract reports a 33% delineation improvement for the surgically targetable tumor and 23% for the complete tumor extent over a publicly trained model.
  • Data size alone did not linearly predict success: public initial vs. preliminary consensus model showed significant improvements on centralized out-of-sample data (p < 7 × 10−18), while preliminary vs. final consensus model differences were insignificant (p > 0.067).
  • The final consensus model was always superior or insignificantly different from an ensemble of local models from the four largest contributing sites (>200 cases each) across all sub-compartments (pAverage = 0.1, pET = 0.5, pTC = 0.2, pWT = 0.06).
  • Federated learning was robust to data quality issues: Site 48 had erroneous annotations (n = 46) and significantly lower local validation scores (pET < 3 × 10−12, pTC < 3 × 10−12, pWT < 3 × 10−12), but the global consensus model performance was not adversely affected.

Methods

  • Design: staged real-world federated learning federation using a central aggregation server, starting from a public initial model (231 cases from 16 sites), then a preliminary consensus model (2,471 cases from 35 sites), and finally a final consensus model (6,314 cases from 71 sites).
  • Data: multi-parametric MRI (T1, T1Gd, T2, T2-FLAIR) from adult-type diffuse glioma/glioblastoma patients; 55 independent collaborating sites contributed 6,083 cases, with 49 sites in training (5,493 cases) and 6 out-of-sample sites (590 cases; centralized out-of-sample subset n = 332). Each site held out 20% of cases as local validation.
  • Analysis: federated averaging (FedAvg) of model updates, 3D-ResUNet architecture, Dice similarity coefficient (DSC) for ET, TC, and WT, Wilcoxon signed-rank tests, and 73 federated rounds.

Limitations

  • The paper states there is no automated mechanism to assess inputted data quality from collaborators, which could result in models trained on sub-optimal data; it also states that only data quality issues largely affecting the global validation score could be identified during training, while subtler issues were found only after training.
  • The paper states it used a single off-the-shelf neural network architecture, whereas model ensembles have been shown to perform better for this task, and it remains unexplored how such a strategy would work in a federated study.
  • The paper states that the federation required significant manual coordination and orchestration, with no tools available to automate it, and that security/privacy threats remain; evident from the text, the study relied on a network of trusted sites, manual inspection of local and global validation scores, and TLS plus trusted execution environments considered sufficient for its threat model.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-022-33407-5

Cites

116 of 116 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 111 cited sources checked · 0 supported · 0 flagged

  • Findingreasoning

    The average Dice similarity coefficient across all three tumor sub-compartments per case and across all cases of all sites was 0.66 for the public initial model.

    “To summarize the model performance with a single collective score, we then calculate the average DSC (across all 3 tumor sub-compartments per case, and then across all cases of all sites) as equal to 0.66.”
    Quote in the paperSubstantial claim
  • Findingown data

    The largest and most significant performance improvement (33%, p < 7 × 10−60) was noted for the tumor core (TC) sub-compartment.

    “Notably, the largest and most significant improvement (33%, p < 7 × 10−60) is noted for the TC sub-compartment, which is surgically actionable and not a trivial sub-compartment to delineate accurately43,44.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [43] Beiko, J. et al. Idh1 mutant malignant astrocytomas are more amenable to surgical resection and have a survival benefit associated with maximal surgical resection. Neuro-oncology 16, 81–91 (2014). source

      Not checked in the free run.

    • cites [44] Olson, J. J. Congress of neurological surgeons systematic review and evidence-based guidelines for the treatment of adults with progressive glioblastoma update: introduction and methods. J. Neuro-oncol 158, 133–137 (2022). source

      Not checked in the free run.

  • Findingown data

    Due to the lack of generalizable solutions to quantify site contributions, quality issues were only identified after the model training.

    “As noted in our results, due to the lack of such generalizable solutions, we were only able to identify quality issues after the model training.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The dataset used in this study is the largest and most diverse dataset of glioblastoma patients ever considered in the literature.

    “Notably, this describes the largest and most diverse dataset of glioblastoma patients ever considered in the literature.”
    • Also stated in: § Discussion · plain
      “Notably, this extensive global footprint of the collaborating sites in this study also yields the largest dataset ever reported in the literature assessing this rare disease.”
    Quote in the paperSubstantial claim
  • Findingown data

    The final consensus model performance is always superior or insignificantly different (pAverage = 0.1, pET = 0.5, pTC = 0.2, pWT = 0.06) to the ensemble of the local models of the four largest contributing collaborators, for all tumor sub-compartments.

    “The findings of this evaluation indicate that the final consensus model performance is always superior or insignificantly different (pAverage = 0.1, pET = 0.5, pTC = 0.2, pWT = 0.06, Wilcoxon signed-rank test) to the ensemble of the local models of these four largest contributing collaborators, for all tumor sub-compartments (Fig. 2).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The independent models and their ensemble from the four largest sites could have inherited some of the public initial model's data diversity because they initiated training from it, which could justify the observed insignificant differences.
    • Rests on an assumption: The centralized out-of-sample subset of 332 cases is representative of the complete out-of-sample data population of 590 cases.
  • Classified as probably not a substantial claim. 49
  • May combine two claims. 46
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Triplet models are three times more expensive than singlet models during model inference.

    “However, the decision of using a singlet or a triplet model should also rely on computational cost considerations, as triplets will be three times more expensive than singlets during model inference.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On centralized out-of-sample data, there were major significant performance improvements between the public initial model and the preliminary consensus model (p < 7 × 10−18), but insignificant improvements between the preliminary and final consensus models (p > 0.067), for all sub-compartments and their average.

    “Specifically, we noted major significant (p < 7 × 10−18, Wilcoxon signed-rank test) performance improvements between the public initial model and the preliminary consensus model, as opposed to the insignificant (p > 0.067, Wilcoxon signed-rank test) ones between the preliminary and the final consensus model, as quantified in the centralized out-of-sample data for all sub-compartments and their ave”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The centralized out-of-sample subset of 332 cases is representative of the complete out-of-sample data population of 590 cases.

  • May combine two claims.

    Findingown data

    The curves of Fig. 3a indicate that the global consensus model continues to consistently gain knowledge from the federation as a whole during training, highlighting robustness to such data quality issues.

    “The curves of Fig. 3a indicate that the global consensus model continues to consistently gain knowledge from the federation as a whole during training, highlighting robustness to such data quality issues.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Only data quality issues that largely affected the global validation score could be identified and corrected during training, while those with more subtle effects were identified only after training by looking for relatively low local validation scores across collaborating sites.

    “However, only data quality issues that largely affected the global validation score could be identified and corrected during training. Those with more subtle effects in the global validation score were only identified after the completion of the model training by looking for relatively low local validation scores of the consensus model across collaborating sites.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The final consensus model achieved significant performance improvements over the public initial model on complete out-of-sample data of 15% for enhancing tumor (ET), 27% for tumor core (TC), and 16% for whole tumor (WT).

    “To further evaluate the potential generalizability improvements of the final consensus model on unseen data, we compared it with the public initial model against the complete out-of-sample data and noted significant performance improvements of 15% (pET < 1 × 10−5), 27% (pTC < 1 × 10−16), and 16% (pWT < 1 × 10−7), for ET, TC, and WT, respectively (Fig. 1d).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Only small differences are observed between the singlet and triplet consensus models, and these differences diminish as the sub-compartment size increases.

    “Only small differences are observed between the singlet and triplet models, and these differences diminish as the sub-compartment size increases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Comparing the means of singlet and triplet models, the larger and only significant performance improvement difference compared to the public initial model is noted for the enhancing tumor (ET) sub-compartment (improved by <3%, pET = 0.02), followed by tumor core (TC) (improved by <1.4%, pTC = 0.09), and then whole tumor (WT) (improved by <1.1%, pWT = 0.2).

    “Comparing the means of singlet and triplet, the larger (and only significant) performance improvement difference compared to the public initial model is noted for the ET sub-compartment (improved by < 3%, pET = 0.02), followed by TC (improved by < 1.4%, pTC = 0.09), and then lastly WT (improved by < 1.1%, pWT = 0.2) (Tables S1 and S2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The final consensus model achieved significant performance improvements over the public initial model on collaborators' local validation data of 27% for enhancing tumor (ET), 33% for tumor core (TC), and 16% for whole tumor (WT).

    “Following model training across all sites, the final consensus model garnered significant performance improvements against the collaborators’ local validation data of 27% (pET < 1 × 10−36), 33% (pTC < 1 × 10−59), and 16% (pWT < 1 × 10−21), for ET, TC, and WT, respectively (Fig. 1c).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The final consensus model was trained on over twice the number of cases compared to the preliminary consensus model, and included two of the largest contributing sites (Sites 1 and 4).

    “This happened even though the final consensus model was trained on over twice the number of cases (and included 2 of the largest contributing sites—Sites 1 and 4) when compared to the preliminary consensus model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The data from all 55 collaborating sites followed a male:female ratio of 1.47:1 with ages ranging between 7 and 94 years.

    “The data from all 55 collaborating sites followed a male:female ratio of 1.47:1 with ages ranging between 7 and 94 years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the post-study survey, 88% of sites used a dedicated workstation for their local workload, and the remaining 12% used either a containerized form of the FeTS tool or a virtual machine.

    “In terms of hardware requirements at each site, 88% used a dedicated workstation for their local workload, and the remaining 12% used either a containerized form of the FeTS tool or a virtual machine.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    55 sites participated as independent collaborators in the study, defining a dataset of 6083 cases.

    “55 sites participated as independent collaborators in the study defining a dataset of 6083 cases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the post-study survey, 84% of participating sites appreciated the user-friendly interface of the provided tool and its associated complete functionality, and indicated their intention to use it and recommend it for projects and data analysis pipelines beyond the scope of this study.

    “Furthermore, 84% of the sites appreciated the user-friendly interface of the provided tool and its associated complete functionality (beyond its FL backend), and indicated their intention to use it and recommend it for projects and data analysis pipelines beyond the scope of this study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The study demonstrates a 33% delineation improvement for the surgically targetable tumor and 23% for the complete tumor extent over a publicly trained model.

    “We demonstrate a 33% delineation improvement for the surgically targetable tumor, and 23% for the complete tumor extent, over a publicly trained model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The out-of-sample data totaled 590 cases intentionally held out of the federation, with the intention of validating the consensus model in completely unseen cases.

    “The out-of-sample data totaled 590 cases intentionally held out of the federation, with the intention of validating the consensus model in completely unseen cases.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the post-study survey, 86% of collaborating sites used either the FeTS Tool, CaPTk, or ITK-SNAP to generate reference standard labels for their local data, while the remaining 14% used either 3D-Slicer, the BraTS toolkit, or something else.

    “To generate the reference standard labels for their local data, 86% of the collaborating sites indicated that they used either the FeTS Tool39 (i.e., the tool developed for this study), CaPTk51, or ITK-SNAP52, whereas the remaining 14% used either 3D-Slicer53, the BraTS toolkit54, or something else.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the post-study survey, 96% of participating sites found the comprehensive documentation on preprocessing and data curation essential and thought that lack of such documentation could have resulted in inconsistent annotations.

    “According to the provided feedback, 96% of the sites found the comprehensive documentation on preprocessing and data curation essential and thought that lack of such documentation could have resulted in inconsistent annotations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The instantiation of the federation involved a significant amount of coordination between each site, and at the time there were no tools available to automate such coordination and orchestration.

    “Moreover, the instantiation of the federation involved a significant amount of coordination between each site and considering the limited real-world FL studies at the time, there were no tools available to automate such coordination and orchestration.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Following screening for the availability of the four required mpMRI scans with sufficient signal-to-noise ratio judged by visual observation, a subset of 362 cases from the original trial data were included in this study.

    “Following screening for the availability of the four required mpMRI scans with sufficient signal-to-noise ratio judged by visual observation, a subset of 362 cases from the original trial data were included in this study.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    The MRI scanners used for data acquisition were from multiple vendors (Siemens, GE, Philips, Hitachi, Toshiba), with magnetic field strength ranging from 1T to 3T.

    “The MRI scanners used for data acquisition were from multiple vendors (i.e., Siemens, GE, Philips, Hitachi, Toshiba), with magnetic field strength ranging from 1T to 3T.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The collaborative network of the study spans 6 continents, with data from 71 geographically distinct sites.

    “The collaborative network of the present study spans 6 continents (Fig. 1), with data from 71 geographically distinct sites.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the post-study survey, 92% of participating sites found the documentation relating to establishing secure connectivity to the aggregation server easy to follow and essential to expedite reviews by the related groups.

    “Additionally, 92% found the documentation relating to establishing secure connectivity to the aggregation server easy to follow and essential to expedite reviews by the related groups.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The run-time optimizations increased throughput by 4.48× (equal to the speedup improvement, since batch size was 1).

    “higher throughput (equal to the 4.48 × speedup improvement since the batch size used is equal to 1)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationown data

    The finding that increased data improves performance reinforces the importance of using large and diverse data for generalizable machine learning models to ultimately drive patient care.

    “The conclusion of this finding reinforces the importance of using large and diverse data for generalizable models to ultimately drive patient care.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    The finding that global consensus model performance is not adversely affected by Site 48 data quality issues supports the importance of robustness at a global scale.

    “This finding supports the importance of robustness at a global scale.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    The four largest sites used for independent model training represent comprehensive cancer centers with affiliated sophisticated labs focusing on brain tumor research, and hence were familiar with the intricacies of computational analyses.

    “Though these findings are an indication that the inclusion of more data alone may not lead to better performance, it is worth noting that these four largest sites used for the independent model training represent comprehensive cancer centers (compared to hospitals in community settings) with affiliated sophisticated labs focusing on brain tumor research, and hence were familiar with the intrica”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Twenty percent of the total cases contributed by each participating site were excluded from the model training process and used as local validation data.

    “To quantitatively evaluate the performance of the trained models, 20% of the total cases contributed by each participating site were excluded from the model training process and used as “local validation data”.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Federated learning provides an alternative paradigm for accurate and generalizable machine learning by only sharing numerical model updates from decentralized data.

    “Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates.”
    • Also stated in: § Introduction · plain
      “In contrast to this centralized paradigm, “federated learning” (FL) describes a paradigm where models are trained by only sharing model parameter updates from decentralized data (i.e., each site retains its data locally)10,11,14–16, without sacrificing performance when compared to CL-trained models11,15,17–21.”
    Quote in the paperSubstantial claim0 of 10 sources checked
    • cites [10] Rieke, N. et al. The future of digital health with federated learning. NPJ Digit. Med. 3, 1–7 (2020). source

      Not checked in the free run.

    • cites [11] Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 1–12 (2020). source

      Not checked in the free run.

    • cites [14] McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (eds Singh, A. & Zhu, J.) 1273–1282 (PMLR, 2017).

      Not checked in the free run.

    • cites [15] Sheller, M. J., Reina, G. A., Edwards, B., Martin, J. & Bakas, S.Multi-institutional deep learning modeling without sharing patient data: a feasibility study on brain tumor segmentation. In International MICCAI Brainlesion Workshop (eds Crimi, A. et al.) 92–104 (Springer, 2018). source

      Not checked in the free run.

    • cites [16] Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021). source

      Not checked in the free run.

    • cites [17] Chang, K. et al. Distributed deep learning networks among institutions for medical imaging. J. Am. Med. Inform. Assoc. 25, 945–954 (2018). source

      Not checked in the free run.

    • cites [18] Nilsson, A., Smith, S., Ulm, G., Gustavsson, E. & Jirstrand, M. A performance evaluation of federated learning algorithms. In Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning, 1–8 (Association for Computing Machinery, New York, 2018). source

      Not checked in the free run.

    • cites [19] Sarma, K. V. et al. Federated learning improves site performance in multicenter deep learning without data sharing. J. Am. Med. Inform. Assoc. 28, 1259–1264 (2021). source

      Not checked in the free run.

    • cites [20] Shen, C. et al. Multi-task federated learning for heterogeneous pancreas segmentation. In Clinical Image-Based Procedures, Distributed and Collaborative Learning, Artificial Intelligence for Combating COVID-19 and Secure and Privacy-Preserving Machine Learning (eds Laura, C. O. et al.) 101–110 (Springer, 2021). source

      Not checked in the free run.

    • cites [21] Yang, D. et al. Federated semi-supervised learning for covid region segmentation in chest ct using multi-national data from China, Italy, Japan. Med. Image Anal. 70, 101992 (2021). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A subset of 332 out-of-sample cases was aggregated to serve as a centralized out-of-sample dataset.

    “To facilitate further evaluation without burdening the collaborating sites, a subset (n = 332) of these cases was aggregated to serve as a “centralized out-of-sample” dataset.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The centralized out-of-sample subset of 332 cases is representative of the complete out-of-sample data population of 590 cases.

  • May combine two claims.

    Premisecitation

    Glioblastoma is the most common malignant primary brain tumor, but it is classified as a rare disease because its incidence rate of 3 per 100,000 people is below the rare disease definition rate of less than 10 per 100,000 people.

    “While glioblastoma is the most common malignant primary brain tumor26–28, it is still classified as a “rare” disease, as its incidence rate (i.e., 3/100,000 people) is substantially lower than the rare disease definition rate (i.e., <10/100,000 people)24.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [26] Brennan, C. W. et al. The somatic genomic landscape of glioblastoma. Cell 155, 462–477 (2013). source

      Not checked in the free run.

    • cites [27] Verhaak, R. G. et al. Integrated genomic analysis identifies clinically relevant subtypes of glioblastoma characterized by abnormalities in pdgfra, idh1, egfr, and nf1. Cancer Cell 17, 98–110 (2010). source

      Not checked in the free run.

    • cites [28] Sottoriva, A. et al. Intratumor heterogeneity in human glioblastoma reflects cancer evolutionary dynamics. Proc. Natl Acad. Sci. USA 110, 4009–4014 (2013). source

      Not checked in the free run.

    • cites [24] Griggs, R. C. et al. Clinical research for rare disease: opportunities, challenges, and solutions. Mol. Genet. Metab. 96, 20–26 (2009). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Six sites were not involved in any of the training stages to represent an unseen out-of-sample data population of 590 cases.

    “To further evaluate the generalizability of the models in unseen data, 6 sites were not involved in any of the training stages to represent an unseen “out-of-sample” data population of 590 cases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The preliminary federation model was already within the intra- and inter-rater variability range for this use-case (20% and 28%, respectively), so any further improvements for the full federation consensus model would be expected to be minimal.

    “Further noting that the preliminary federation model was already within the intra- and inter-rater variability range for this use-case (20% and 28%, respectively)47, any further improvements for the full federation consensus model would be expected to be minimal35–38.”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [47] Mazzara, G. P., Velthuizen, R. P., Pearlman, J. L., Greenberg, H. M. & Wagner, H. Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation. Int. J. Radiat. Oncol.* Biol.* Phys. 59, 300–312 (2004). source

      Not checked in the free run.

    • cites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source

      Not checked in the free run.

    • cites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source

      Not checked in the free run.

    • cites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).

      Not checked in the free run.

    • cites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The local validation scores at Site 48 indicate that the model is not gaining knowledge from these local data, and their comparison with the average scores across the federation indicates that the global consensus model performance is not adversely affected.

    “These local validation scores (Fig. 3b) indicate that the model is not gaining knowledge from these local data, and their comparison with the average scores across the federation (Fig. 3a) indicates that the global consensus model performance is not adversely affected.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Compared with existing real-world federated learning studies, this study's use-case is larger in scale and substantially more complex because it addresses a multi-parametric multi-class problem with reference standards requiring expert manual annotation and requires harmonized preprocessing for MRI acquisition differences.

    “The extended global footprint and the task complexity are what sets this study apart from current literature, since it dealt with a multi-parametric multi-class problem with reference standards that require expert clinicians following an involved manual annotation protocol, rather than simply recording a categorical entry from medical records16,32.”
    • Also stated in: § Discussion · plain
      “In comparison with the limited existing real-world FL studies16,32, our use-case is larger in scale and substantially more complex, since it (1) addresses a multi-parametric multi-class problem, with reference standards that require expert collaborating clinicians to follow an involved manual annotation protocol, rather than simply recording a categorical entry from medical records, and (2) requ”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [16] Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021). source

      Not checked in the free run.

    • cites [32] Roth, H. R. et al. Federated learning for breast density classification: a real-world implementation. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (eds Albarqouni, S. et al.) 181–191 (Springer, 2020).

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Single sites cannot collect large and diverse datasets to train robust and generalizable machine learning models for glioblastoma, necessitating collaboration between geographically distinct sites.

    “This means that single sites cannot collect large and diverse datasets to train robust and generalizable ML models, necessitating collaboration between geographically distinct sites.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A deep learning model trained on 641 glioblastoma cases from 8 sites produced higher quality predictions on average than reference standard labels created by radiology expert operators.

    “These findings are aligned with literature observations (on the same use-case)48, where a DL model49 trained on 641 glioblastoma cases from 8 sites produced higher quality predictions on average than those created as reference standard labels by radiology expert operators.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [48] Mitchell, J. R. et al. Deep neural network to locate and segment brain tumors outperformed the expert technicians who created the training data. J. Med. Imaging 7, 055501 (2020). source

      Not checked in the free run.

    • cites [49] Kamnitsas, K. et al. Efficient multi-scale 3d CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Model ensembles perform better than a single off-the-shelf neural network for the task of detecting glioblastoma sub-compartment boundaries.

    “Additionally, we used a single off-the-shelf neural network architecture for training, but it has been shown that model ensembles perform better for the task at hand35–38, and it remains to be explored how such a strategy could be explored in a federated study.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source

      Not checked in the free run.

    • cites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source

      Not checked in the free run.

    • cites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).

      Not checked in the free run.

    • cites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The study and results presented in this manuscript comply with all relevant ethical regulations and follow appropriate ethical standards.

    “The study and results presented in this manuscript comply with all relevant ethical regulations and follow appropriate ethical standards in conducting research and writing the manuscript, following all applicable laws and regulations regarding the treatment of human subjects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The study was based on a collaborative network of trusted sites, with authentication based on personal communication across collaborating sites and the combination of TLS and TEEs considered sufficient.

    “Our study was based on a collaborative network of trusted sites, where authentication was based on personal communication across collaborating sites and the combination of TLS and TEEs were considered sufficient.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The collaborative network consists of trusted sites.
    • The authors state this as an assumption: The combination of TLS and TEEs is sufficient to address the security and privacy concerns for this study.

  • May combine two claims.

    Interpretationreasoning

    Federated learning can offer an alternative to centralized learning, potentially creating a paradigm shift that alleviates the need for data sharing, and hence increase access to geographically distinct collaborators, thereby increasing the size and diversity of data used to train machine learning models.

    “Thus, FL can offer an alternative to CL, potentially creating a paradigm shift that alleviates the need for data sharing, and hence increase access to geographically distinct collaborators, thereby increasing the size and diversity of data used to train ML models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Homomorphic encryption, secure multiparty compute, and trusted execution environments allow collaborative computations to be performed with untrusted parties while maintaining confidentiality of the inputs.

    “Homomorphic encryption58, secure multiparty compute59, and trusted execution environments (TEEs)60,61 allow for collaborative computations to be performed with untrusted parties while maintaining confidentiality of the inputs to the computation.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [58] Gentry, C. Fully homomorphic encryption using ideal lattices. In Proc. 41st Annual ACM Symposium on Theory of Computing, 169–178 (Association for Computing Machinery, New York, 2009). source

      Not checked in the free run.

    • cites [59] Yao, A. C.Protocols for secure computations. In 23rd Annual Symposium on Foundations of Computer Science (SFCS 1982), 160–164 (IEEE, 1982). source

      Not checked in the free run.

    • cites [60] Sabt, M., Achemlal, M. & Bouabdallah, A. Trusted execution environment: what it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/ISPA Vol. 1, 57–64 (IEEE, 2015). source

      Not checked in the free run.

    • cites [61] Schneider, M., Masti, R. J., Shinde, S., Capkun, S. & Perez, R.Sok: Hardware-supported trusted execution environments. arXiv preprint arXiv:2205.12742 (2022).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One of the most common types of MRI noise is based on the inhomogeneity of the magnetic field.

    “One of the most common types of MRI noise is based on the inhomogeneity of the magnetic field83.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [83] Song, S., Zheng, Y. & He, Y. A review of methods for bias correction in medical images. Biomed. Eng. Rev. 1, 2375–9151 (2017). source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The 6 out-of-sample sites were allocated based on their availability, i.e., they indicated expected delayed participation, rendering them optimal for model generalizability validation.

    “These specific 6 out-of-sample sites (Site IDs: 8, 11, 19, 20, 21, 43) were allocated based on their availability, i.e., they have indicated expected delayed participation rendering them optimal for model generalizability validation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 6 out-of-sample sites are representative enough for model generalizability validation.

  • Classified as probably not a substantial claim.

    Premisecitation

    Starting from a pre-trained model leads to faster convergence.

    “The public initial model was used to initialize the FL training, instead of a randomly generated initialization, as starting from a pre-trained model leads to faster convergence41.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [41] Young, J. C. & Suryadibrata, A. Applicability of various pre-trained deep convolutional neural networks for pneumonia classification based on x-ray images. Int. J. Adv. Trends Comput. Sci. Eng. 9, 2649–2654 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The models generated during this study have the potential to make a far-reaching clinical impact because glioblastoma boundary detection is critical for treatment planning and the requisite first step for further quantitative analyses.

    “Since glioblastoma boundary detection is critical for treatment planning and the requisite first step for further quantitative analyses, the models generated during this study have the potential to make a far-reaching clinical impact.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Different security and privacy threats remain for federated learning, including attempted extraction of training data information from models, model theft, and submission of poison model updates.

    “These threats include attempted extraction of training data information from intermediate and final models, model theft, and submission of poison model updates with the goal of introducing unwanted model behavior (including incentivizing the model to memorize more information about the training data in support of subsequent extraction, i.e., leakage).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [55] Kairouz, P. et al. Advances and open problems in federated learning. Found. Trends® in Mach. Learn. 14, 1–210 (2021). source

      Not checked in the free run.

    • cites [56] Nasr, M., Shokri, R. & Houmansadr, A. Comprehensive privacy analysis of deep learning: passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE Symposium on Security and Privacy (SP), 739–753 (IEEE, 2019). source

      Not checked in the free run.

    • cites [57] Lam, M., Wei, G.-Y., Brooks, D., Reddi, V. J. & Mitzenmacher, M. Gradient disaggregation: breaking privacy in federated learning by reconstructing the user participant matrix. In International Conference on Machine Learning, 5959–5968 (PMLR, 2021).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Each security and privacy technology for federated learning has its own benefits and costs or limitations, such as increased computational complexity, associated hardware requirements, and reduced quality of computational output.

    “Each of these technologies comes with its own benefits in terms of security and/or privacy, as well as costs and limitations, such as increased computational complexity, associated hardware requirements and/or reduced quality of computational output”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Use of the private retrospective data collection of each collaborating site was approved by their respective institutional review board, and informed consent from all participants was obtained and stored.

    “Use of the private retrospective data collection of each collaborating site has been approved by their respective institutional review board, where informed consent from all participants was also obtained and stored.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Assurance that remote computations are executed with integrity can be designed for with hardware-based trust provided by TEEs, as well as with some software-based integrity checking.

    “Finally, assurance that remote computations are executed with integrity can be designed for with the use of hardware-based trust provided by TEEs, as well as with some software-based integrity checking65.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [65] Tramer, F. & Boneh, D. Slalom: fast, verifiable and private execution of neural networks in trusted hardware. arXiv preprint arXiv:1806.03287 (2018).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    There is no automated mechanism to assess inputted data quality from collaborators, which could result in models trained using sub-optimal data.

    “Although our study has the potential to become the baseline upon which future ML research studies will be done, there is no automated mechanism to assess inputted data quality from collaborators, which could result in models trained using sub-optimal data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Federated learning has tremendous potential in healthcare, particularly towards addressing health disparities, under-served populations, and rare diseases, by enabling machine learning models to gain knowledge from ample and diverse data that would otherwise not be available.

    “FL has tremendous potential in healthcare22,23, particularly towards addressing health disparities, under-served populations, and “rare” diseases24, by enabling ML models to gain knowledge from ample and diverse data that would otherwise not be available.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [22] De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342–1350 (2018). source

      Not checked in the free run.

    • cites [23] Hannun, A. Y. et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat. Med. 25, 65–69 (2019). source

      Not checked in the free run.

    • cites [24] Griggs, R. C. et al. Clinical research for rare disease: opportunities, challenges, and solutions. Mol. Genet. Metab. 96, 20–26 (2009). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    T1-FLAIR scans were intentionally excluded to avoid mixing varying tissue appearance due to the type of sequence across native T1-weighted scans.

    “The only exclusion criterion was for T1-FLAIR scans that were intentionally excluded to avoid mixing varying tissue appearance due to the type of sequence, across native T1-weighted scans.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    From all 55 collaborating sites, 49 were chosen for the training phase, and 6 sites were categorized as out-of-sample, none of which were part of the training stage.

    “From all 55 collaborating sites, 49 were chosen to be part of the training phase, and 6 sites were categorized as “out-of-sample”, i.e., none of these were part of the training stage.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Eligibility of collaborating sites to participate in the federation was determined based on data availability and approval by their respective institutional review board.

    “The eligibility of collaborating sites to participate in the federation was determined based on data availability, and approval by their respective institutional review board.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Differentially private training algorithms mitigate information leakage from both collaborator model updates and global consensus aggregated models.

    “Differentially private training algorithms62–64 allow for mitigation of information leakage from both the collaborator model updates and the global consensus aggregated models.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [62] Dwork, C. Differential privacy: a survey of results. In International Conference on Theory and Applications of Models of Computation 1–19 (Springer, 2008). source

      Not checked in the free run.

    • cites [63] Wei, K. et al. Federated learning with differential privacy: algorithms and performance analysis. IEEE Trans. Inf. Forensics Secur. 15, 3454–3469 (2020). source

      Not checked in the free run.

    • cites [64] Adnan, M., Kalra, S., Cresswell, J. C., Taylor, G. W. & Tizhoosh, H. R. Federated learning and differential privacy for medical image analysis. Sci. Rep. 12, 1–10 (2022). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    BrainMaGe was explicitly developed to address brain scans in the presence of diffuse glioma and considers brain shape as a prior, making it agnostic to the sequence/modality input.

    “For this step we utilized the Brain Mask Generator (BrainMaGe)87, which has been explicitly developed to address brain scans in presence of diffuse glioma and considers brain shape as a prior, hence being agnostic to the sequence/modality input.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [87] Thakur, S. et al. Brain extraction on MRI scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training. NeuroImage 220, 117081 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The BraTS challenge provides the largest publicly available and manually-curated mpMRI dataset of diffuse glioma patients and a harmonized preprocessing pipeline to handle inter-site acquisition protocol differences.

    “The BraTS challenge35–38, seeking methodological advancements in the domain of neuro-oncology, has been providing the community with (i) the largest publicly available and manually-curated mpMRI dataset of diffuse glioma patients (an example of which is illustrated in Fig. 1b), and (ii) a harmonized preprocessing pipeline51,70,71 to handle differences in inter-site acquisition protocols.”
    Quote in the paperNot a substantial claim0 of 7 sources checked
    • cites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source

      Not checked in the free run.

    • cites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source

      Not checked in the free run.

    • cites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).

      Not checked in the free run.

    • cites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

      Not checked in the free run.

    • cites [51] Davatzikos, C. et al. Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome. J. Med. Imaging 5, 011018 (2018). source

      Not checked in the free run.

    • cites [70] Rathore, S. et al. Brain cancer imaging phenomics toolkit (brain-captk): an interactive platform for quantitative analysis of glioblastoma. In International MICCAI Brainlesion Workshop (eds Crimi, A. et al.) 133–145 (Springer, 2017). source

      Not checked in the free run.

    • cites [71] Pati, S. et al. The cancer imaging phenomics toolkit (captk): technical overview. In International MICCAI Brainlesion Workshop (eds Crimi, A. & Bakas, S.) 380–394 (Springer, 2019). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Cases with any of the four required mpMRI sequences (T1, T1Gd, T2, T2-FLAIR) missing were not included in the study.

    “Cases with any of these sequences missing were not included in the study.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To avoid potential data leakage, the BraTS dataset used for the public initial model was reduced by removing cases acquired by sites that also participated in this study, resulting in 231 cases from 16 international sites.

    “To avoid any potential data leakage, we reduced the size of the complete BraTS dataset by removing cases acquired by these specific sites, resulting in a dataset of 231 cases from 16 international sites, with varying contributing cases across sites (Fig. 1e).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The data considered in this study described patient populations with adult-type diffuse glioma, specifically displaying the radiological features of glioblastoma, scanned with multiparametric MRI.

    “The data considered in this study described patient populations with adult-type diffuse glioma30, and specifically displaying the radiological features of glioblastoma, scanned with mpMRI to characterize the anatomical tissue structure25.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [30] Louis, D. N. et al. The 2021 WHO classification of tumors of the central nervous system: a summary. Neuro-oncology 23, 1231–1251 (2021). source

      Not checked in the free run.

    • cites [25] Shukla, G. et al. Advanced magnetic resonance imaging in glioblastoma: a review. Chin. Clin. Oncol. 6, 40 (2017). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Aggregation of all collaborator updates using federated averaging (FedAvg), weighted according to each collaborator's contributing data, was chosen based on a prior performance evaluation for this use-case.

    “Based on our previously conducted performance evaluation for this use-case11, we chose to perform aggregation of all collaborator updates in the present study, using the federated averaging (FedAvg) approach14, i.e., average of collaborator’s model updates weighted according to collaborator’s contributing data.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [11] Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 1–12 (2020). source

      Not checked in the free run.

    • cites [14] McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (eds Singh, A. & Zhu, J.) 1273–1282 (PMLR, 2017).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    Accurate detection of glioblastoma sub-compartment boundaries is of particular relevance for neurosurgical and radiotherapy planning in patients with this disease.

    “We have demonstrated the utility of an FL workflow to develop an accurate and generalizable ML model for detecting glioblastoma sub-compartment boundaries, a finding which is of particular relevance for neurosurgical and radiotherapy planning in patients with this disease.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    This study is the first federated learning study at this scale and task complexity.

    “As this study is the first at this scale and task complexity, we have compiled a set of governance insights from our experience that can serve as considerations for future successful FL studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The performance of every model was compared against the local validation data cohort on every federated round.

    “The performance of every model was compared against the local validation data cohort on every federated round.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    All operations on the aggregator were performed on secure hardware (TEE) to increase trust by all parties in the confidentiality of the model updates and confidence in the integrity of the computations.

    “Additionally, we performed all operations on the aggregator on secure hardware (TEE112), in order to increase the trust by all parties in the confidentiality of the model updates being computed and shared, as well as to increase the confidence in the integrity of the computations being performed113.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [112] Kaissis, G. A., Makowski, M. R., Rückert, D. & Braren, R. F. Secure, privacy-preserving and federated machine learning in medical imaging. Nat. Mach. Intell. 2, 305–311 (2020). source

      Not checked in the free run.

    • cites [113] Ekberg, J.-E., Kostiainen, K. & Asokan, N. The untapped potential of trusted execution environments on mobile devices. IEEE Secur. Priv. 12, 29–37 (2014). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationcitation

    Ensuring strict compliance with the preprocessing and annotation protocol for the generation of reference standards was vital for the model to learn correct information during training.

    “Following the precise definition of our problem statement35–38, ensuring strict compliance with the preprocessing and annotation protocol for the generation of reference standards was vital for the model to learn correct information during training.”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source

      Not checked in the free run.

    • cites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source

      Not checked in the free run.

    • cites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).

      Not checked in the free run.

    • cites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    The governance insights from this study differ from previous literature that describes federated learning studies that were smaller in scale and involved simpler tasks.

    “These insights differ from previous literature that describes studies that were smaller in scale and involved simpler tasks16,32.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [16] Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021). source

      Not checked in the free run.

    • cites [32] Roth, H. R. et al. Federated learning for breast density classification: a real-world implementation. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (eds Albarqouni, S. et al.) 181–191 (Springer, 2020).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Running an ensemble of multiple models is typically done in academic research projects.

    “which is typically done in academic research projects35–38”
    • Also stated in: § Model runtime in low-resource settings · plain
      “as typically done in academic research projects35–38”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source

      Not checked in the free run.

    • cites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source

      Not checked in the free run.

    • cites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).

      Not checked in the free run.

    • cites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    All network communications during the FL model training process were based on TLS to mitigate potential exposure of information during transit.

    “In this study, all the network communications during the FL model training process were based on TLS111, to mitigate potential exposure of information during transit.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [111] Knauth, T. et al. Integrating remote attestation with transport layer security. arXiv preprint arXiv:1801.05863 (2018).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The security discussions highlighted the benefit of the platform being open-source, making security code reviews easier.

    “These security discussions highlighted the benefit of the platform being open-source, making security code reviews easier.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Once data were preprocessed, training and validation cohorts were created randomly in a 4:1 ratio, and the splits were preserved during the entire duration of the FL training to prevent data leakage.

    “Once the data were preprocessed, training and validation cohorts were created randomly in a 4:1 ratio, and the splits were preserved during the entire duration of the FL training to prevent data leakage.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The feasibility of TEEs for federated training was demonstrated by running the aggregator workload on secure enclaves of Intel SGX hardware, which ensured the confidentiality of the updates being aggregated and the integrity of the consensus model.

    “Additionally, we demonstrated the feasibility of TEEs60,61 for federated training by running the aggregator workload on the secure enclaves of Intel’s Secure Guard Extensions (SGX) hardware (Intel® Xeon® E-2286M vPro 8-Core 2.4-5.0GHz Turbo), which ensured the confidentiality of the updates being aggregated and the integrity of the consensus model.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [60] Sabt, M., Achemlal, M. & Bouabdallah, A. Trusted execution environment: what it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/ISPA Vol. 1, 57–64 (IEEE, 2015). source

      Not checked in the free run.

    • cites [61] Schneider, M., Masti, R. J., Shinde, S., Capkun, S. & Perez, R.Sok: Hardware-supported trusted execution environments. arXiv preprint arXiv:2205.12742 (2022).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    After not observing any meaningful changes since round 42, the training was stopped after a total of 73 federated rounds.

    “After not observing any meaningful changes since round 42, we stopped the training after a total of 73 federated rounds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Debugging issues with inputted local data and annotations is more difficult during federated learning due to the level of coordination and/or privacy issues involved, since the data are always retained at the collaborating site.

    “In general, debugging issues with the inputted local data and annotations is more difficult during FL due to the level of coordination and/or privacy issues involved, since the data are always retained at the collaborating site.”
    Quote in the paperNot a substantial claim
74 of 74

  • The authors state this as an assumption.

    The independent models and their ensemble from the four largest sites could have inherited some of the public initial model's data diversity because they initiated training from it, which could justify the observed insignificant differences.

    “The underlying assumption for these results is that since each of these collaborators initiated their training from the public initial model (which included diverse data from 16 sites), their independent models and their ensemble could have inherited some of the initial model’s data diversity, which could justify the observed insignificant differences (Fig. 2 and Supplementary Fig. 3).”

  • The paper relies on this without support.

    The centralized out-of-sample subset of 332 cases is representative of the complete out-of-sample data population of 590 cases.

    “To facilitate further evaluation without burdening the collaborating sites, a subset (n = 332) of these cases was aggregated to serve as a “centralized out-of-sample” dataset.”

  • The paper relies on this without support.

    The collaborative network consists of trusted sites.

    “Our study was based on a collaborative network of trusted sites, where authentication was based on personal communication across collaborating sites and the combination of TLS and TEEs were considered sufficient.”

  • The authors state this as an assumption.

    The combination of TLS and TEEs is sufficient to address the security and privacy concerns for this study.

    “Our study was based on a collaborative network of trusted sites, where authentication was based on personal communication across collaborating sites and the combination of TLS and TEEs were considered sufficient.”

  • The paper relies on this without support.

    The 6 out-of-sample sites are representative enough for model generalizability validation.

    “These specific 6 out-of-sample sites (Site IDs: 8, 11, 19, 20, 21, 43) were allocated based on their availability, i.e., they have indicated expected delayed participation rendering them optimal for model generalizability validation.”

  • The paper relies on this without support.

    Manual refinements by neuroradiology experts following the annotation protocol produce accurate reference standard labels.

    “It was communicated to all participating sites to leverage the annotations generated using the automated mechanism as a baseline on which manual refinements were needed by neuroradiology experts, following a consistently communicated annotation protocol.”

  • The paper relies on this without support.

    Clinical environments typically have constrained computational resources, such as limited availability of specialized hardware (e.g., DL acceleration cards) and increased memory, which affect the runtime performance of deep learning inference workloads.

    “Clinical environments typically have constrained computational resources, such as the availability of specialized hardware (e.g., DL acceleration cards) and increased memory, which affect the runtime performance of DL inference workloads.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For rare cancers, because individual sites may treat few patients, multisite data integration is necessary to produce meaningful scientific or clinical insights.

    “It is even more acutely important for rare cancers, where individual sites may only treat a small number of such patients, and multisite data integration is necessary to establish meaningful scientific or clinical insights (161).”

    From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [161]

  • Pati and colleagues used federated learning across 71 sites to predict tumor boundaries in multiparametric MRI scans of glioblastoma patients.

    “A study by Pati and colleagues (161) also used FL across 71 sites to predict tumor boundaries within multiparametric MRI scans of glioblastoma patients.”

    From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [161]

  • Centralized learning, in which sites transfer data to a central secure enclave after legal agreements and security protocols, is the standard default approach in most commercial and academic AI efforts to date.

    “This is the standard, default approach used in most commercial and academic AI efforts to date.”

    From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [161]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Whole-tumor improvement is reported as both 23% and 16%

    Major

    The abstract reports a 23% improvement for the complete tumor extent (WT), while the Results report 16% for WT on both local validation and complete out-of-sample data.

  • Centralized out-of-sample sample size differs

    Minor

    The Methods define the centralized out-of-sample dataset as 332 cases, whereas the Fig. 2 caption gives 154 cases for an evaluation described as being on that dataset.

  • Complete out-of-sample sample size differs

    Minor

    The Methods define the held-out out-of-sample data as 590 cases, but the Fig. 1 caption labels the complete out-of-sample evaluation as 518 cases. No further exclusion is stated for the latter figure.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (eds Singh, A. & Zhu, J.) 1273–1282 (PMLR, 2017).
  2. Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018). · arXiv
  3. Raghu, M., Zhang, C., Kleinberg, J. & Bengio, S. Transfusion: Understanding transfer learning for medical imaging. Proceedings of the 33rd International Conference on Neural Information Processing Systems 32, 3347–3357 (Association for Computing Machinery, 2019).
  4. Marsland, S. Novelty detection in learning systems. Neural Comput. Surv. 3, 157–195 (2003).
  5. Lam, M., Wei, G.-Y., Brooks, D., Reddi, V. J. & Mitzenmacher, M. Gradient disaggregation: breaking privacy in federated learning by reconstructing the user participant matrix. In International Conference on Machine Learning, 5959–5968 (PMLR, 2021).
  6. Mustra, M., Delac, K. & Grgic, M. Overview of the dicom standard. In 2008 50th International Symposium ELMAR Vol. 1, 39–44 (IEEE, 2008).
  7. Cox, R. et al. A (sort of) new image data format standard: Nifti-1. In: Proc. 10th Annual Meeting of the Organization for Human Brain Mapping 22 (Wiley, 2004).
  8. Barrodale, I. L1 approximation and the analysis of data. J. R. Stat. Soc.: Ser. C (Appl. Stat.) 17, 51–57 (1968).
  9. Rodriguez, A. et al. Lower numerical precision deep learning inference and training. Intel White Paper 3, 1–19 (2018).
  10. Lin, D., Talathi, S. & Annapureddy, S. Fixed point quantization of deep convolutional networks. In International Conference on Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 2849–2858 (PMLR, 2016).

Cited by

Sorted by citations

Related

Authors
Pati, S., Baid, U., Edwards, B., Sheller, M., Wang, S.-H., Reina, G. A., Foley, P., Gruzdev, A., Karkada, D., Davatzikos, C., Sako, C., Ghodasara, S., Bilello, M., Mohan, S., Vollmuth, P., Brugnara, G., Preetha, C. J., Sahm, F., Maier-Hein, K., Zenk, M., Bendszus, M., Wick, W., Calabrese, E., Rudie, J., Villanueva-Meyer, J., Cha, S., Ingalhalikar, M., Jadhav, M., Pandey, U., Saini, J., Garrett, J., Larson, M., Jeraj, R., Currie, S., Frood, R., Fatania, K., Huang, R. Y., Chang, K., Balaña, C., Capellades, J., Puig, J., Trenkler, J., Pichler, J., Necker, G., Haunschmidt, A., Meckel, S., Shukla, G., Liem, S., Alexander, G. S., Lombardo, J., Palmer, J. D., Flanders, A. E., Dicker, A. P., Sair, H. I., Jones, C. K., Venkataraman, A., Jiang, M., So, T. Y., Chen, C., Heng, P. A., Dou, Q., Kozubek, M., Lux, F., Michálek, J., Matula, P., Keřkovský, M., Kopřivová, T., Dostál, M., Vybíhal, V., Vogelbaum, M. A., Mitchell, J. R., Farinhas, J., Maldjian, J. A., Yogananda, C. G. B., Pinho, M. C., Reddy, D., Holcomb, J., Wagner, B. C., Ellingson, B. M., Cloughesy, T. F., Raymond, C., Oughourlian, T., Hagiwara, A., Wang, C., To, M.-S., Bhardwaj, S., Chong, C., Agzarian, M., Falcão, A. X., Martins, S. B., Teixeira, B. C. A., Sprenger, F., Menotti, D., Lucio, D. R., LaMontagne, P., Marcus, D., Wiestler, B., Kofler, F., Ezhov, I., Metz, M., Jain, R., Lee, M., Lui, Y. W., McKinley, R., Slotboom, J., Radojewski, P., Meier, R., Wiest, R., Murcia, D., Fu, E., Haas, R., Thompson, J., Ormond, D. R., Badve, C., Sloan, A. E., Vadmal, V., Waite, K., Colen, R. R., Pei, L., Ak, M., Srinivasan, A., Bapuraj, J. R., Rao, A., Wang, N., Yoshiaki, O., Moritani, T., Turk, S., Lee, J., Prabhudesai, S., Morón, F., Mandel, J., Kamnitsas, K., Glocker, B., Dixon, L. V. M., Williams, M., Zampakis, P., Panagiotopoulos, V., Tsiganos, P., Alexiou, S., Haliassos, I., Zacharaki, E. I., Moustakas, K., Kalogeropoulou, C., Kardamakis, D. M., Choi, Y. S., Lee, S.-K., Chang, J. H., Ahn, S. S., Luo, B., Poisson, L., Wen, N., Tiwari, P., Verma, R., Bareja, R., Yadav, I., Chen, J., Kumar, N., Smits, M., van der Voort, S. R., Alafandi, A., Incekara, F., Wijnenga, M. M. J., Kapsas, G., Gahrmann, R., Schouten, J. W., Dubbink, H. J., Vincent, A. J. P. E., van den Bent, M. J., French, P. J., Klein, S., Yuan, Y., Sharma, S., Tseng, T.-C., Adabi, S., Niclou, S. P., Keunen, O., Hau, A.-C., Vallières, M., Fortin, D., Lepage, M., Landman, B., Ramadass, K., Xu, K., Chotai, S., Chambless, L. B., Mistry, A., Thompson, R. C., Gusev, Y., Bhuvaneshwar, K., Sayah, A., Bencheqroun, C., Belouali, A., Madhavan, S., Booth, T. C., Chelliah, A., Modat, M., Shuaib, H., Dragos, C., Abayazeed, A., Kolodziej, K., Hill, M., Abbassy, A., Gamal, S., Mekhaimar, M., Qayati, M., Reyes, M., Park, J. E., Yun, J., Kim, H. S., Mahajan, A., Muzi, M., Benson, S., Beets-Tan, R. G. H., Teuwen, J., Herrera-Trujillo, A., Trujillo, M., Escobar, W., Abello, A., Bernal, J., Gómez, J., Choi, J., Baek, S., Kim, Y., Ismael, H., Allen, B., Buatti, J. M., Kotrotsou, A., Li, H., Weiss, T., Weller, M., Bink, A., Pouymayou, B., Shaykh, H. F., Saltz, J., Prasanna, P., Shrestha, S., Mani, K. M., Payne, D., Kurc, T., Pelaez, E., Franco-Maldonado, H., Loayza, F., Quevedo, S., Guevara, P., Torche, E., Mendoza, C., Vera, F., Ríos, E., López, E., Velastin, S. A., Ogbole, G., Soneye, M., Oyekunle, D., Odafe-Oyibotha, O., Osobu, B., Shu’aibu, M., Dorcas, A., Dako, F., Simpson, A. L., Hamghalam, M., Peoples, J. J., Hu, R., Tran, A., Cutler, D., Moraes, F. Y., Boss, M. A., Gimpel, J., Veettil, D. K., Schmidt, K., Bialecki, B., Marella, S., Price, C., Cimino, L., Apgar, C., Shah, P., Menze, B., Barnholtz-Sloan, J. S., Martin, J., & Bakas, S.
Affiliations
  • Technical University of Munich (DE)
  • University of Pennsylvania (US)
  • Intel (United States) (US)
  • University Hospital Heidelberg (DE)
  • German Cancer Research Center (DE)
  • University of California, San Francisco (US)
  • Symbiosis International University (IN)
  • National Institute of Mental Health and Neurosciences (IN)
  • University of Wisconsin–Madison (US)
  • Leeds Teaching Hospitals NHS Trust (GB)
  • Brigham and Women's Hospital (US)
  • Massachusetts General Hospital (US)
  • Parc de Salut (ES)
  • Institut d'Investigació Biomèdica de Girona (ES)
  • Kepler Universitätsklinikum (AT)
  • Christiana Care Health System (US)
  • Thomas Jefferson University (US)
  • University of Maryland, Baltimore (US)
  • The Ohio State University Comprehensive Cancer Center – Arthur G. James Cancer Hospital and Richard J. Solove Research Institute (US)
  • Johns Hopkins University (US)
  • Chinese University of Hong Kong (HK)
  • Masaryk University (CZ)
  • Moffitt Cancer Center (US)
  • University of Alberta (CA)
  • The University of Texas Southwestern Medical Center (US)
  • University of California, Los Angeles (US)
  • Flinders University (AU)
  • Flinders Medical Centre (AU)
  • Baylor College of Medicine (US)
  • Universidade Estadual de Campinas (UNICAMP) (BR)
  • Federal Institute of São Paulo (BR)
  • Hospital de Clínicas Universidade Federal do Paraná (BR)
  • Universidade Federal do Paraná (BR)
  • Washington University in St. Louis (US)
  • TUM Klinikum (DE)
  • New York University (US)
  • University of Bern (CH)
  • University of Colorado Anschutz (US)
  • University Hospitals of Cleveland (US)
  • University Hospitals Seidman Cancer Center (US)
  • Case Western Reserve University (US)
  • National Institutes of Health (US)
  • The University of Texas MD Anderson Cancer Center (US)
  • University of Pittsburgh Medical Center (US)
  • University of Pittsburgh (US)
  • University of Michigan (US)
  • University of Oxford (GB)
  • Imperial College London (GB)
  • Imperial College Healthcare NHS Trust (GB)
  • University of Patras (GR)
  • Yonsei University (KR)
  • Henry Ford Health (US)
  • Alberta Machine Intelligence Institute (CA)
  • Erasmus MC (NL)
  • Erasmus MC Cancer Institute (NL)
  • Icahn School of Medicine at Mount Sinai (US)
  • Luxembourg Institute of Health (LU)
  • Université de Sherbrooke (CA)
  • Vanderbilt University (US)
  • Vanderbilt University Medical Center (US)
  • Georgetown University (US)
  • MedStar Georgetown University Hospital (US)
  • King's College London (GB)
  • Queen's University (CA)
  • Stoke Mandeville Hospital (GB)
  • Cairo University (EG)
  • Asan Medical Center (KR)
  • Clatterbridge Cancer Centre NHS Foundation Trust (GB)
  • University of Washington (US)
  • The Netherlands Cancer Institute (NL)
  • Universidad del Valle (CO)
  • University of Iowa (US)
  • University of Zurich (CH)
  • University of Alabama at Birmingham (US)
  • Stony Brook University (US)
  • Sociedad de Lucha Contra el Cáncer del Ecuador (EC)
  • Escuela Superior Politecnica del Litoral (EC)
  • University of Concepción (CL)
  • University College Hospital, Ibadan (NG)
  • Muhammad Abdullahi Wase Teaching Hospital (NG)
  • Qazvin Islamic Azad University (IR)
  • American College of Radiology (US)
Corresponding author
Bakas, Spyridon
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2022-12-05
Volume
13
Issue
1
Pages
7346-7346
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-022-33407-5
PMC
PMC9722782
OpenAlex
W4310852955
Semantic Scholar
6c783c7aa7d6832fae4dd54ed523b8cc84d2833a
Cited by
394
References
115
Citations by year
2026: 111 · 2025: 114 · 2024: 99 · 2023: 62 · 2022: 8
FWCI
23.29
Type
journal article
Language
en
Update notices
correction (2023-01-26)
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
  • YUHSpace (Yonsei University Medical Library) (submittedVersion, cc-by-nc-nd)
  • Open Repository and Bibliography (University of Luxembourg) (submittedVersion, cc-by)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
  • Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
  • Research Portal (King's College London) (submittedVersion, cc-by)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
  • Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) (submittedVersion, cc-by)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • National Science Foundation (2040462, 2040532)
  • U.S. Department of Defense (200025, U10CA180820, W81XWH, R50CA211270, W81XWH-18, U10CA180794)
  • V Foundation for Cancer Research
  • Intel Corporation
  • Brigham and Women's Hospital
  • Massachusetts General Hospital
  • Radiological Society of North America (RR2011)
  • American College of Radiology Imaging Network (U10CA180820, CA079778, CA080098)
  • University of Pennsylvania
  • Ohio State University
  • Varian Medical Systems (UL1TR001433)
  • Washington University in St. Louis
  • Henry Ford Health System
  • Canadian Institute for Advanced Research
  • Johns Hopkins University
  • University of Pittsburgh
  • Case Western Reserve University
  • Thomas Jefferson University
  • Case Comprehensive Cancer Center, Case Western Reserve University
  • School of Medicine, Case Western Reserve University
  • Deutsches Krebsforschungszentrum
  • University of Patras
  • University of Bern
  • Moffitt Cancer Center
  • Wellcome Trust
  • Alberta Machine Intelligence Institute
  • Ohio State University Comprehensive Cancer Center – Arthur G. James Cancer Hospital and Richard J. Solove Research Institute
  • UK Research and Innovation
  • Inselspital, Universitätsspital Bern
  • Kepler Universitätsklinikum
  • National Institute of Mental Health and Neurosciences
  • Agenția Națională pentru Cercetare și Dezvoltare
  • University of Alberta
  • Cancer Research UK (28832, 9RO1/1403, C19942/A28832)
  • Imperial College London
  • University of Oxford
  • European Commission (316679, 757173, HORIZON2020, FP7- PEOPLE-2012-ITN)
  • Deutsche Forschungsgemeinschaft (SFB824, SFB 1389, 404521405)
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (203148, 140958)
  • Flinders University
  • Fundação de Amparo à Pesquisa do Estado de São Paulo (2014/, FB0008, 2014/12236- 1)
  • Fonds National de la Recherche Luxembourg (C20/BM/14646004/GLASSLUX)
  • Yonsei University
  • Ministerstvo Zdravotnictví Ceské Republiky (NU21‐08‐00359)
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico (303808/2018‐7, FB0008, 2014/12236-1)
  • KWF Kankerbestrijding (EMCR 2015–7859)
  • Chinese University of Hong Kong
  • Shanghai Jiao Tong University
  • Technische Universität München
  • Centre d'Imagerie BioMédicale
  • Universidade Federal do Paraná
  • Masarykova Univerzita
  • Lékařská fakulta, Masarykova univerzita
  • Deutschen Konsortium für Translationale Krebsforschung
  • Ruijin Hospital
  • Universitätsklinikum Heidelberg
  • Agencia Nacional de Investigación y Desarrollo (BASAL/FB0008, FB0008, FB210017)
  • National Institutes of Health (UL1TR001409, UL1TR001433, 5u24ca215109-05, 3u10ca079778-14s1, W81XWH, SFB 824, 1s10od023495-01, RSG-16-005-01, CA180868, CA180820, CA180794, NCI:U24CA215109)
  • University of California, Los Angeles
  • University of Texas MD Anderson Cancer Center
  • Medical Center, University of Pittsburgh
  • Perelman School of Medicine, University of Pennsylvania
  • Anschutz Medical Campus, University of Colorado
  • Whiting School of Engineering, Johns Hopkins University
  • NYU Grossman School of Medicine
  • Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital
  • Medical Research Council (MR/W021684/1, C19942/A28832, 2096671)
  • Engineering and Physical Sciences Research Council (203148/ Z/16/Z)
  • Yonsei University College of Medicine
  • School of Medicine, Shanghai Jiao Tong University
  • National Institute of Mental Health
  • National Cancer Institute (U01CA079778, U01CA248226, P30 CA-51008, U10-CA-21661, U10CA180820, U10CA037422, CA 079778, U10CA080098, U01CA176110, CA 080098, CA180820, CA180822, CA180794, U01CA080098, U10CA180868, U10CA37422, R37CA214955, R01CA233888, U10 CA079778, U10CA021661, U10CA180794, U24CA189523, U24CA215109,, U01CA242871, R01CA082500, CA180868, P50CA211015, P30CA051008, R01CA269948, R50CA211270, R01CA270027, U10CA180822)
  • National Institute of Neurological Disorders and Stroke (R01NS042645, NINDS:R01NS042645)
  • National Institute of Biomedical Imaging and Bioengineering (R21EB030209)
  • U.S. National Library of Medicine (R01LM013151)
  • National Center for Advancing Translational Sciences
  • David Geffen School of Medicine, University of California, Los Angeles
  • FP7 People: Marie-Curie Actions (FP7‐PEOPLE‐2012‐ITN, 2012-ITN)
  • CHIST-ERA
  • DOD Peer Reviewed Cancer Research Program (W81XWH-18-1-0404)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
federated learning, glioblastoma, tumor boundary detection, multi-site collaboration, rare tumor, data privacy, consensus model
MeSH terms
Machine Learning, Big Data, Glioblastoma, Humans, Information Dissemination, Rare Diseases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review