Federated learning enables big data for rare cancer boundary detection

In the paper

“Following the precise definition of our problem statement35–38, ensuring strict compliance with the preprocessing and annotation protocol for the generation of reference standards was vital for the model to learn correct information during training.”

What the sources say

• Partly supported

  The paper’s “Expert annotation of tumor structures” section describes a defined annotation protocol, and “Lessons learned” strongly recommends homogenized preprocessing. It does not say strict compliance with both protocols was vital for a model to learn correctly; “Lessons learned” instead says many algorithms can overcome slight inconsistencies or errors in training data. Page numbers are not available in our copy. (LLM, conf=0.83)

  cites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). sourceExperimental — accuracy not yet measured
• Partly supported

  The Methods (Pre-processing) and Technical Validation (Manual correction) sections describe defined image-processing steps and expert-reviewed corrections to segmentation labels. They do not state that strict compliance with those protocols was vital for a model to learn correct information during training; that appears to be the citing authors’ inference. (LLM, conf=0.82)

  cites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). sourceExperimental — accuracy not yet measured
• Not checked yetcites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).
• Not checked yetcites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

In the paper

“However, the decision of using a singlet or a triplet model should also rely on computational cost considerations, as triplets will be three times more expensive than singlets during model inference.”

In the paper

“Specifically, we noted major significant (p < 7 × 10−18, Wilcoxon signed-rank test) performance improvements between the public initial model and the preliminary consensus model, as opposed to the insignificant (p > 0.067, Wilcoxon signed-rank test) ones between the preliminary and the final consensus model, as quantified in the centralized out-of-sample data for all sub-compartments and their ave”

In the paper

“The curves of Fig. 3a indicate that the global consensus model continues to consistently gain knowledge from the federation as a whole during training, highlighting robustness to such data quality issues.”

In the paper

“However, only data quality issues that largely affected the global validation score could be identified and corrected during training. Those with more subtle effects in the global validation score were only identified after the completion of the model training by looking for relatively low local validation scores of the consensus model across collaborating sites.”

In the paper

“To further evaluate the potential generalizability improvements of the final consensus model on unseen data, we compared it with the public initial model against the complete out-of-sample data and noted significant performance improvements of 15% (pET < 1 × 10−5), 27% (pTC < 1 × 10−16), and 16% (pWT < 1 × 10−7), for ET, TC, and WT, respectively (Fig. 1d).”

In the paper

“Only small differences are observed between the singlet and triplet models, and these differences diminish as the sub-compartment size increases.”

In the paper

“Comparing the means of singlet and triplet, the larger (and only significant) performance improvement difference compared to the public initial model is noted for the ET sub-compartment (improved by < 3%, pET = 0.02), followed by TC (improved by < 1.4%, pTC = 0.09), and then lastly WT (improved by < 1.1%, pWT = 0.2) (Tables S1 and S2).”

In the paper

“Following model training across all sites, the final consensus model garnered significant performance improvements against the collaborators’ local validation data of 27% (pET < 1 × 10−36), 33% (pTC < 1 × 10−59), and 16% (pWT < 1 × 10−21), for ET, TC, and WT, respectively (Fig. 1c).”

In the paper

“This happened even though the final consensus model was trained on over twice the number of cases (and included 2 of the largest contributing sites—Sites 1 and 4) when compared to the preliminary consensus model.”

In the paper

“The data from all 55 collaborating sites followed a male:female ratio of 1.47:1 with ages ranging between 7 and 94 years.”

In the paper

“In terms of hardware requirements at each site, 88% used a dedicated workstation for their local workload, and the remaining 12% used either a containerized form of the FeTS tool or a virtual machine.”

In the paper

“55 sites participated as independent collaborators in the study defining a dataset of 6083 cases.”

In the paper

“Furthermore, 84% of the sites appreciated the user-friendly interface of the provided tool and its associated complete functionality (beyond its FL backend), and indicated their intention to use it and recommend it for projects and data analysis pipelines beyond the scope of this study.”

In the paper

“We demonstrate a 33% delineation improvement for the surgically targetable tumor, and 23% for the complete tumor extent, over a publicly trained model.”

In the paper

“The out-of-sample data totaled 590 cases intentionally held out of the federation, with the intention of validating the consensus model in completely unseen cases.”

In the paper

“To generate the reference standard labels for their local data, 86% of the collaborating sites indicated that they used either the FeTS Tool39 (i.e., the tool developed for this study), CaPTk51, or ITK-SNAP52, whereas the remaining 14% used either 3D-Slicer53, the BraTS toolkit54, or something else.”

In the paper

“According to the provided feedback, 96% of the sites found the comprehensive documentation on preprocessing and data curation essential and thought that lack of such documentation could have resulted in inconsistent annotations.”

In the paper

“Moreover, the instantiation of the federation involved a significant amount of coordination between each site and considering the limited real-world FL studies at the time, there were no tools available to automate such coordination and orchestration.”

In the paper

“Following screening for the availability of the four required mpMRI scans with sufficient signal-to-noise ratio judged by visual observation, a subset of 362 cases from the original trial data were included in this study.”

In the paper

“The MRI scanners used for data acquisition were from multiple vendors (i.e., Siemens, GE, Philips, Hitachi, Toshiba), with magnetic field strength ranging from 1T to 3T.”

In the paper

“The collaborative network of the present study spans 6 continents (Fig. 1), with data from 71 geographically distinct sites.”

In the paper

“Additionally, 92% found the documentation relating to establishing secure connectivity to the aggregation server easy to follow and essential to expedite reviews by the related groups.”

In the paper

“higher throughput (equal to the 4.48 × speedup improvement since the batch size used is equal to 1)”

In the paper

“The conclusion of this finding reinforces the importance of using large and diverse data for generalizable models to ultimately drive patient care.”

In the paper

“This finding supports the importance of robustness at a global scale.”

In the paper

“Though these findings are an indication that the inclusion of more data alone may not lead to better performance, it is worth noting that these four largest sites used for the independent model training represent comprehensive cancer centers (compared to hospitals in community settings) with affiliated sophisticated labs focusing on brain tumor research, and hence were familiar with the intrica”

In the paper

“To quantitatively evaluate the performance of the trained models, 20% of the total cases contributed by each participating site were excluded from the model training process and used as “local validation data”.”

In the paper

“Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates.”

• Also stated in: § Introduction · plain

  “In contrast to this centralized paradigm, “federated learning” (FL) describes a paradigm where models are trained by only sharing model parameter updates from decentralized data (i.e., each site retains its data locally)10,11,14–16, without sacrificing performance when compared to CL-trained models11,15,17–21.”

What the sources say

• Not checked yetcites [10] Rieke, N. et al. The future of digital health with federated learning. NPJ Digit. Med. 3, 1–7 (2020). source
• Not checked yetcites [11] Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 1–12 (2020). source
• Not checked yetcites [14] McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (eds Singh, A. & Zhu, J.) 1273–1282 (PMLR, 2017).
• Not checked yetcites [15] Sheller, M. J., Reina, G. A., Edwards, B., Martin, J. & Bakas, S.Multi-institutional deep learning modeling without sharing patient data: a feasibility study on brain tumor segmentation. In International MICCAI Brainlesion Workshop (eds Crimi, A. et al.) 92–104 (Springer, 2018). source
• Not checked yetcites [16] Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021). source
• Not checked yetcites [17] Chang, K. et al. Distributed deep learning networks among institutions for medical imaging. J. Am. Med. Inform. Assoc. 25, 945–954 (2018). source
• Not checked yetcites [18] Nilsson, A., Smith, S., Ulm, G., Gustavsson, E. & Jirstrand, M. A performance evaluation of federated learning algorithms. In Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning, 1–8 (Association for Computing Machinery, New York, 2018). source
• Not checked yetcites [19] Sarma, K. V. et al. Federated learning improves site performance in multicenter deep learning without data sharing. J. Am. Med. Inform. Assoc. 28, 1259–1264 (2021). source
• Not checked yetcites [20] Shen, C. et al. Multi-task federated learning for heterogeneous pancreas segmentation. In Clinical Image-Based Procedures, Distributed and Collaborative Learning, Artificial Intelligence for Combating COVID-19 and Secure and Privacy-Preserving Machine Learning (eds Laura, C. O. et al.) 101–110 (Springer, 2021). source
• Not checked yetcites [21] Yang, D. et al. Federated semi-supervised learning for covid region segmentation in chest ct using multi-national data from China, Italy, Japan. Med. Image Anal. 70, 101992 (2021). source

In the paper

“To facilitate further evaluation without burdening the collaborating sites, a subset (n = 332) of these cases was aggregated to serve as a “centralized out-of-sample” dataset.”

In the paper

“While glioblastoma is the most common malignant primary brain tumor26–28, it is still classified as a “rare” disease, as its incidence rate (i.e., 3/100,000 people) is substantially lower than the rare disease definition rate (i.e., <10/100,000 people)24.”

What the sources say

• Not checked yetcites [26] Brennan, C. W. et al. The somatic genomic landscape of glioblastoma. Cell 155, 462–477 (2013). source
• Not checked yetcites [27] Verhaak, R. G. et al. Integrated genomic analysis identifies clinically relevant subtypes of glioblastoma characterized by abnormalities in pdgfra, idh1, egfr, and nf1. Cancer Cell 17, 98–110 (2010). source
• Not checked yetcites [28] Sottoriva, A. et al. Intratumor heterogeneity in human glioblastoma reflects cancer evolutionary dynamics. Proc. Natl Acad. Sci. USA 110, 4009–4014 (2013). source
• Not checked yetcites [24] Griggs, R. C. et al. Clinical research for rare disease: opportunities, challenges, and solutions. Mol. Genet. Metab. 96, 20–26 (2009). source

In the paper

“To further evaluate the generalizability of the models in unseen data, 6 sites were not involved in any of the training stages to represent an unseen “out-of-sample” data population of 590 cases.”

In the paper

“Further noting that the preliminary federation model was already within the intra- and inter-rater variability range for this use-case (20% and 28%, respectively)47, any further improvements for the full federation consensus model would be expected to be minimal35–38.”

What the sources say

• Not checked yetcites [47] Mazzara, G. P., Velthuizen, R. P., Pearlman, J. L., Greenberg, H. M. & Wagner, H. Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation. Int. J. Radiat. Oncol.* Biol.* Phys. 59, 300–312 (2004). source
• Not checked yetcites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source
• Not checked yetcites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source
• Not checked yetcites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).
• Not checked yetcites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

In the paper

“These local validation scores (Fig. 3b) indicate that the model is not gaining knowledge from these local data, and their comparison with the average scores across the federation (Fig. 3a) indicates that the global consensus model performance is not adversely affected.”

In the paper

“The extended global footprint and the task complexity are what sets this study apart from current literature, since it dealt with a multi-parametric multi-class problem with reference standards that require expert clinicians following an involved manual annotation protocol, rather than simply recording a categorical entry from medical records16,32.”

• Also stated in: § Discussion · plain

  “In comparison with the limited existing real-world FL studies16,32, our use-case is larger in scale and substantially more complex, since it (1) addresses a multi-parametric multi-class problem, with reference standards that require expert collaborating clinicians to follow an involved manual annotation protocol, rather than simply recording a categorical entry from medical records, and (2) requ”

What the sources say

• Not checked yetcites [16] Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021). source
• Not checked yetcites [32] Roth, H. R. et al. Federated learning for breast density classification: a real-world implementation. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (eds Albarqouni, S. et al.) 181–191 (Springer, 2020).

In the paper

“This means that single sites cannot collect large and diverse datasets to train robust and generalizable ML models, necessitating collaboration between geographically distinct sites.”

In the paper

“These findings are aligned with literature observations (on the same use-case)48, where a DL model49 trained on 641 glioblastoma cases from 8 sites produced higher quality predictions on average than those created as reference standard labels by radiology expert operators.”

What the sources say

• Not checked yetcites [48] Mitchell, J. R. et al. Deep neural network to locate and segment brain tumors outperformed the expert technicians who created the training data. J. Med. Imaging 7, 055501 (2020). source
• Not checked yetcites [49] Kamnitsas, K. et al. Efficient multi-scale 3d CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017). source

In the paper

“Additionally, we used a single off-the-shelf neural network architecture for training, but it has been shown that model ensembles perform better for the task at hand35–38, and it remains to be explored how such a strategy could be explored in a federated study.”

What the sources say

• Not checked yetcites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source
• Not checked yetcites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source
• Not checked yetcites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).
• Not checked yetcites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

In the paper

“The study and results presented in this manuscript comply with all relevant ethical regulations and follow appropriate ethical standards in conducting research and writing the manuscript, following all applicable laws and regulations regarding the treatment of human subjects.”

In the paper

“Our study was based on a collaborative network of trusted sites, where authentication was based on personal communication across collaborating sites and the combination of TLS and TEEs were considered sufficient.”

In the paper

“Thus, FL can offer an alternative to CL, potentially creating a paradigm shift that alleviates the need for data sharing, and hence increase access to geographically distinct collaborators, thereby increasing the size and diversity of data used to train ML models.”

In the paper

“Homomorphic encryption58, secure multiparty compute59, and trusted execution environments (TEEs)60,61 allow for collaborative computations to be performed with untrusted parties while maintaining confidentiality of the inputs to the computation.”

What the sources say

• Not checked yetcites [58] Gentry, C. Fully homomorphic encryption using ideal lattices. In Proc. 41st Annual ACM Symposium on Theory of Computing, 169–178 (Association for Computing Machinery, New York, 2009). source
• Not checked yetcites [59] Yao, A. C.Protocols for secure computations. In 23rd Annual Symposium on Foundations of Computer Science (SFCS 1982), 160–164 (IEEE, 1982). source
• Not checked yetcites [60] Sabt, M., Achemlal, M. & Bouabdallah, A. Trusted execution environment: what it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/ISPA Vol. 1, 57–64 (IEEE, 2015). source
• Not checked yetcites [61] Schneider, M., Masti, R. J., Shinde, S., Capkun, S. & Perez, R.Sok: Hardware-supported trusted execution environments. arXiv preprint arXiv:2205.12742 (2022).

In the paper

“One of the most common types of MRI noise is based on the inhomogeneity of the magnetic field83.”

What the sources say

• Not checked yetcites [83] Song, S., Zheng, Y. & He, Y. A review of methods for bias correction in medical images. Biomed. Eng. Rev. 1, 2375–9151 (2017). source

In the paper

“These specific 6 out-of-sample sites (Site IDs: 8, 11, 19, 20, 21, 43) were allocated based on their availability, i.e., they have indicated expected delayed participation rendering them optimal for model generalizability validation.”

In the paper

“The public initial model was used to initialize the FL training, instead of a randomly generated initialization, as starting from a pre-trained model leads to faster convergence41.”

What the sources say

• Not checked yetcites [41] Young, J. C. & Suryadibrata, A. Applicability of various pre-trained deep convolutional neural networks for pneumonia classification based on x-ray images. Int. J. Adv. Trends Comput. Sci. Eng. 9, 2649–2654 (2020). source

In the paper

“Since glioblastoma boundary detection is critical for treatment planning and the requisite first step for further quantitative analyses, the models generated during this study have the potential to make a far-reaching clinical impact.”

In the paper

“These threats include attempted extraction of training data information from intermediate and final models, model theft, and submission of poison model updates with the goal of introducing unwanted model behavior (including incentivizing the model to memorize more information about the training data in support of subsequent extraction, i.e., leakage).”

What the sources say

• Not checked yetcites [55] Kairouz, P. et al. Advances and open problems in federated learning. Found. Trends® in Mach. Learn. 14, 1–210 (2021). source
• Not checked yetcites [56] Nasr, M., Shokri, R. & Houmansadr, A. Comprehensive privacy analysis of deep learning: passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE Symposium on Security and Privacy (SP), 739–753 (IEEE, 2019). source
• Not checked yetcites [57] Lam, M., Wei, G.-Y., Brooks, D., Reddi, V. J. & Mitzenmacher, M. Gradient disaggregation: breaking privacy in federated learning by reconstructing the user participant matrix. In International Conference on Machine Learning, 5959–5968 (PMLR, 2021).

In the paper

“Each of these technologies comes with its own benefits in terms of security and/or privacy, as well as costs and limitations, such as increased computational complexity, associated hardware requirements and/or reduced quality of computational output”

In the paper

“Use of the private retrospective data collection of each collaborating site has been approved by their respective institutional review board, where informed consent from all participants was also obtained and stored.”

In the paper

“Finally, assurance that remote computations are executed with integrity can be designed for with the use of hardware-based trust provided by TEEs, as well as with some software-based integrity checking65.”

What the sources say

• Not checked yetcites [65] Tramer, F. & Boneh, D. Slalom: fast, verifiable and private execution of neural networks in trusted hardware. arXiv preprint arXiv:1806.03287 (2018).

In the paper

“Although our study has the potential to become the baseline upon which future ML research studies will be done, there is no automated mechanism to assess inputted data quality from collaborators, which could result in models trained using sub-optimal data.”

In the paper

“FL has tremendous potential in healthcare22,23, particularly towards addressing health disparities, under-served populations, and “rare” diseases24, by enabling ML models to gain knowledge from ample and diverse data that would otherwise not be available.”

What the sources say

• Not checked yetcites [22] De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342–1350 (2018). source
• Not checked yetcites [23] Hannun, A. Y. et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat. Med. 25, 65–69 (2019). source
• Not checked yetcites [24] Griggs, R. C. et al. Clinical research for rare disease: opportunities, challenges, and solutions. Mol. Genet. Metab. 96, 20–26 (2009). source

In the paper

“The only exclusion criterion was for T1-FLAIR scans that were intentionally excluded to avoid mixing varying tissue appearance due to the type of sequence, across native T1-weighted scans.”

In the paper

“From all 55 collaborating sites, 49 were chosen to be part of the training phase, and 6 sites were categorized as “out-of-sample”, i.e., none of these were part of the training stage.”

In the paper

“The eligibility of collaborating sites to participate in the federation was determined based on data availability, and approval by their respective institutional review board.”

In the paper

“Differentially private training algorithms62–64 allow for mitigation of information leakage from both the collaborator model updates and the global consensus aggregated models.”

What the sources say

• Not checked yetcites [62] Dwork, C. Differential privacy: a survey of results. In International Conference on Theory and Applications of Models of Computation 1–19 (Springer, 2008). source
• Not checked yetcites [63] Wei, K. et al. Federated learning with differential privacy: algorithms and performance analysis. IEEE Trans. Inf. Forensics Secur. 15, 3454–3469 (2020). source
• Not checked yetcites [64] Adnan, M., Kalra, S., Cresswell, J. C., Taylor, G. W. & Tizhoosh, H. R. Federated learning and differential privacy for medical image analysis. Sci. Rep. 12, 1–10 (2022). source

In the paper

“For this step we utilized the Brain Mask Generator (BrainMaGe)87, which has been explicitly developed to address brain scans in presence of diffuse glioma and considers brain shape as a prior, hence being agnostic to the sequence/modality input.”

What the sources say

• Not checked yetcites [87] Thakur, S. et al. Brain extraction on MRI scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training. NeuroImage 220, 117081 (2020). source

In the paper

“The BraTS challenge35–38, seeking methodological advancements in the domain of neuro-oncology, has been providing the community with (i) the largest publicly available and manually-curated mpMRI dataset of diffuse glioma patients (an example of which is illustrated in Fig. 1b), and (ii) a harmonized preprocessing pipeline51,70,71 to handle differences in inter-site acquisition protocols.”

What the sources say

• Not checked yetcites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source
• Not checked yetcites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source
• Not checked yetcites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).
• Not checked yetcites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).
• Not checked yetcites [51] Davatzikos, C. et al. Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome. J. Med. Imaging 5, 011018 (2018). source
• Not checked yetcites [70] Rathore, S. et al. Brain cancer imaging phenomics toolkit (brain-captk): an interactive platform for quantitative analysis of glioblastoma. In International MICCAI Brainlesion Workshop (eds Crimi, A. et al.) 133–145 (Springer, 2017). source
• Not checked yetcites [71] Pati, S. et al. The cancer imaging phenomics toolkit (captk): technical overview. In International MICCAI Brainlesion Workshop (eds Crimi, A. & Bakas, S.) 380–394 (Springer, 2019). source

In the paper

“Cases with any of these sequences missing were not included in the study.”

In the paper

“To avoid any potential data leakage, we reduced the size of the complete BraTS dataset by removing cases acquired by these specific sites, resulting in a dataset of 231 cases from 16 international sites, with varying contributing cases across sites (Fig. 1e).”

In the paper

“The data considered in this study described patient populations with adult-type diffuse glioma30, and specifically displaying the radiological features of glioblastoma, scanned with mpMRI to characterize the anatomical tissue structure25.”

What the sources say

• Not checked yetcites [30] Louis, D. N. et al. The 2021 WHO classification of tumors of the central nervous system: a summary. Neuro-oncology 23, 1231–1251 (2021). source
• Not checked yetcites [25] Shukla, G. et al. Advanced magnetic resonance imaging in glioblastoma: a review. Chin. Clin. Oncol. 6, 40 (2017). source

In the paper

“Based on our previously conducted performance evaluation for this use-case11, we chose to perform aggregation of all collaborator updates in the present study, using the federated averaging (FedAvg) approach14, i.e., average of collaborator’s model updates weighted according to collaborator’s contributing data.”

What the sources say

• Not checked yetcites [11] Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 1–12 (2020). source
• Not checked yetcites [14] McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (eds Singh, A. & Zhu, J.) 1273–1282 (PMLR, 2017).

In the paper

“We have demonstrated the utility of an FL workflow to develop an accurate and generalizable ML model for detecting glioblastoma sub-compartment boundaries, a finding which is of particular relevance for neurosurgical and radiotherapy planning in patients with this disease.”

In the paper

“As this study is the first at this scale and task complexity, we have compiled a set of governance insights from our experience that can serve as considerations for future successful FL studies.”

In the paper

“The performance of every model was compared against the local validation data cohort on every federated round.”

In the paper

“Additionally, we performed all operations on the aggregator on secure hardware (TEE112), in order to increase the trust by all parties in the confidentiality of the model updates being computed and shared, as well as to increase the confidence in the integrity of the computations being performed113.”

What the sources say

• Not checked yetcites [112] Kaissis, G. A., Makowski, M. R., Rückert, D. & Braren, R. F. Secure, privacy-preserving and federated machine learning in medical imaging. Nat. Mach. Intell. 2, 305–311 (2020). source
• Not checked yetcites [113] Ekberg, J.-E., Kostiainen, K. & Asokan, N. The untapped potential of trusted execution environments on mobile devices. IEEE Secur. Priv. 12, 29–37 (2014). source

In the paper

“These insights differ from previous literature that describes studies that were smaller in scale and involved simpler tasks16,32.”

What the sources say

• Not checked yetcites [16] Dayan, I. et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nat. Med. 27, 1735–1743 (2021). source
• Not checked yetcites [32] Roth, H. R. et al. Federated learning for breast density classification: a real-world implementation. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (eds Albarqouni, S. et al.) 181–191 (Springer, 2020).

In the paper

“which is typically done in academic research projects35–38”

• Also stated in: § Model runtime in low-resource settings · plain

  “as typically done in academic research projects35–38”

What the sources say

• Not checked yetcites [35] Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34, 1993–2024 (2014). source
• Not checked yetcites [36] Bakas, S. et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. data 4, 1–13 (2017). source
• Not checked yetcites [37] Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018).
• Not checked yetcites [38] Baid, U. et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021).

In the paper

“In this study, all the network communications during the FL model training process were based on TLS111, to mitigate potential exposure of information during transit.”

What the sources say

• Not checked yetcites [111] Knauth, T. et al. Integrating remote attestation with transport layer security. arXiv preprint arXiv:1801.05863 (2018).

In the paper

“These security discussions highlighted the benefit of the platform being open-source, making security code reviews easier.”

In the paper

“Once the data were preprocessed, training and validation cohorts were created randomly in a 4:1 ratio, and the splits were preserved during the entire duration of the FL training to prevent data leakage.”

In the paper

“Additionally, we demonstrated the feasibility of TEEs60,61 for federated training by running the aggregator workload on the secure enclaves of Intel’s Secure Guard Extensions (SGX) hardware (Intel® Xeon® E-2286M vPro 8-Core 2.4-5.0GHz Turbo), which ensured the confidentiality of the updates being aggregated and the integrity of the consensus model.”

What the sources say

• Not checked yetcites [60] Sabt, M., Achemlal, M. & Bouabdallah, A. Trusted execution environment: what it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/ISPA Vol. 1, 57–64 (IEEE, 2015). source
• Not checked yetcites [61] Schneider, M., Masti, R. J., Shinde, S., Capkun, S. & Perez, R.Sok: Hardware-supported trusted execution environments. arXiv preprint arXiv:2205.12742 (2022).

In the paper

“After not observing any meaningful changes since round 42, we stopped the training after a total of 73 federated rounds.”

In the paper

“In general, debugging issues with the inputted local data and annotations is more difficult during FL due to the level of coordination and/or privacy issues involved, since the data are always retained at the collaborating site.”