CoChat
Sign Up
Cited by 842Machine Learning in HealthcareCOVID-19 diagnosis using AIArtificial Intelligence in Healthcare and Education

Federated learning for predicting clinical outcomes in patients with COVID-19

Open access
Dayan, Ittai, Roth, Holger R., Zhong, Aoxiao, Harouni, Ahmed, Gentili, Amilcare, Abidin, Anas Z., Liu, Andrew, Costa, Anthony Beardsworth +91 more·2021·Nature Medicine
DOI 10.1038/s41591-021-01506-3PMC PMC9157510

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Federated learning (FL) is a method used for training artificial intelligence models with data from multiple sources while maintaining data anonymity, thus removing many barriers to data sharing. Here we used data from 20 institutes across the globe to train a FL model, called EXAM (electronic medical record (EMR) chest X-ray AI model), that predicts the future oxygen requirements of symptomatic patients with COVID-19 using inputs of vital signs, laboratory data and chest X-rays. EXAM achieved an average area under the curve (AUC) >0.92 for predicting outcomes at 24 and 72 h from the time of initial presentation to the emergency room, and it provided 16% improvement in average AUC measured across all participating sites and an average increase in generalizability of 38% when compared with models trained at a single site using that site's data. For prediction of mechanical ventilation treatment or death at 24 h at the largest independent test site, EXAM achieved a sensitivity of 0.950 and specificity of 0.882. In this study, FL facilitated rapid data science collaboration without data exchange and generated a model that generalized across heterogeneous, unharmonized datasets for prediction of clinical outcomes in patients with COVID-19, setting the stage for the broader use of FL in healthcare.

Integrity signals

Registry

Crossref · 10.1038/s41591-021-01506-3

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 40 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    For client sites with relatively small datasets, the best federated model outperformed both the local model and models trained on larger datasets from five Boston-area sites.

    “In the case of client sites with relatively small datasets, the best FL model markedly outperformed the ‘local’ model as well as models trained on larger datasets from 5 client-sites in the Boston area”
    Quote in the paperSubstantial claim
  • Findingown data

    Federated training increased generalizability by 38% on average compared with single-site models, from 0.667 to 0.920.

    “It also resulted in a 38% generalizability improvement (as defined by the average-AUC when running the model on all test sets; from 0.667 to 0.920 or 25.3 percentage points) of the best global model for predicting 24h oxygen treatment compared to models trained only on a site’s own data”
    Quote in the paperSubstantial claim
  • Findingown data

    For 72-hour mechanical ventilation prediction, EXAM had a false-negative rate of 7.1% at CDH.

    “For MV at 72h, EXAM had a low false-negative rate of 7.1% at CDH.”
    Quote in the paperSubstantial claim
  • Findingown data

    NCH had no cases of mechanical ventilation or death at 24 hours.

    “NCH did not have any cases with MV / death at 24h.”
    Quote in the paperNot a substantial claim
  • Findingown data

    EXAM reached an average AUC above 0.92 for predicting oxygen outcomes at 24 and 72 hours after initial emergency room presentation.

    “EXAM achieved an average area under the curve (AUC) greater than 0.92 for predicting outcomes at 24 and 72h from the time of initial presentation to the ER”
    Quote in the paperSubstantial claim
  • May combine two claims. 31
  • Classified as probably not a substantial claim. 20
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    At independent validation sites, EXAM achieved average AUC 0.944 for 24-hour prediction and 0.924 for 72-hour prediction, exceeding the average performance among training sites.

    “The FL global trained model, EXAM, achieved an average AUC of 0.944 and 0.924 for 24/72h prediction tasks respectively (Table 2 b), which exceeded the average performance among sites used in training EXAM.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Representative false-negative cases at CDH involved missing EMR features and/or chest X-ray motion artifact.

    “showing two false-negative cases from CDH where one case had many missing EMR data features, and the other case had a CXR with motion artifact and some missing EMR features.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For 72-hour oxygen treatment prediction, the global federated model improved average performance by 18% over local models and generalizability by 34%.

    “For the prediction results of 72h oxygen treatments, the best global model training resulted in an average performance improvement of 18% compared to locally trained models, while generalizability of the global model improved on average by 34%”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At unbalanced client-site #16, the federated model had a higher true positive rate for severe cases and a lower false positive rate than the local model.

    “The FL model achieved a higher true positive rate for the two positive (severe) cases and a markedly lower false positive rate compared to the local model”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The federated learning model performed significantly better than locally trained models for predicting 24-hour and 72-hour oxygen support requirements, with a one-sided p-value much less than 1e-3 in both cases.

    “We conducted a Wilcoxon signed-rank test to confirm the significance of the observed improvement in performance between the locally trained model and the FL model for the 24 and 72 hr time point (see Fig. 2 and Extended Data Fig. 1). The null hypothesis was rejected with a one-sided p-value << 1e-3 in both cases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The F-values for local models at five sites were 245.7, 253.4, 342.3, and 389.8, while the federated learning model had an F-value of 843.5.

    “Our results show that the F-values of 5 different local sites are 245.7, 253.4, 342.3, 389.8, while the F-value of the FL model is 843.5.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At MGB, a CORISK threshold A for safe discharge had 99.5% negative predictive value, and threshold B for ICU admission had 50.3% positive predictive value.

    “Histogram of CORISK results at MGB, with an illustration of how the score can be used for patient triage, in which ‘A’ is an example threshold for safe discharge that has 99.5% negative predictive value, and ‘B’ is an example threshold for Intensive Care Unit (ICU) admission that has 50.3% positive predictive value.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The federated learning global model improved average performance by 18% (from 0.760 to 0.899) and average generalizability by 34% (from 0.669 to 0.899) compared to locally trained models.

    “The average performance improved by 18% (from 0.760 to 0.899 or 13.9 percentage points) compared to locally trained models alone, while average generalizability of the global model improved by 34% (from 0.669 to 0.899 or 23.0 percentage points).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    For both 24-hour and 72-hour models, the random splits for the three repeated local and federated experiments were independently generated.

    “For both the 24h and 72h outcome prediction models, the random splits for each of the three repeated local and FL training and evaluation experiments were independently generated.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each client-site randomly split its dataset into 70% training, 10% validation, and 20% testing.

    “Each client-site was asked to randomly split its dataset into 3 parts, 70% for training, 10% for validation, and 20% for testing.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    EXAM was tested at three independent validation sites without retraining, using the model only for validation.

    “The model was not re-trained at these sites, and it was only used for validation purposes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The number of negative patients included was limited to at most 95% of total cases at each client-site.

    “we limited the number of negative patients included to, at –most, 95% of the total cases at each client-site.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    All procedures followed Declaration of Helsinki and Good Clinical Practice principles and were approved by the relevant institutional review boards.

    “All procedures were conducted in accordance with principles for human experimentation as defined in the Declaration of Helsinki and International Conference on Harmonization Good Clinical Practice guidelines and approved by the relevant institutional review boards”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Data across sites were not harmonized before extraction, and the authors did not perform meticulous harmonization of input data.

    “Data between sites was not harmonized prior to being extracted, and in light of real-life clinical informatics circumstances, a meticulous harmonization of the data input was not conducted by the authors”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The MI-CLAIM reporting guidelines for clinical AI models were followed.

    “MI-CLAIM guidelines for reporting of clinical AI models were followed”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    EXAM's design was constrained to objective predictors, unlike many published studies that used subjective clinical impressions.

    “The model’s design was constrained to objective predictors, unlike many published studies that leveraged subjective clinical impressions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Federated learning has shown promise in recent medical imaging applications, including COVID-19 analysis.

    “FL has already shown promise in recent medical imaging applications40–43, including in COVID-19 analysis8,44,45.”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [40] Roth, H. R. et al. Federated learning for breast density classification: a real-world implementation. In Proc. Second MICCAI Workshop, DART 2020 and First MICCAI Workshop, DCL 2020, Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning (eds. Albarqouni, S. et al.) Vol. 12,444, 181–191 (Springer International Publishing, 2020).

      Not checked in the free run.

    • cites [41] Sheller, M. J. et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10, 12598 (2020). source

      Not checked in the free run.

    • cites [42] Remedios, S. W., Butman, J. A., Landman, B. A. & Pham, D. L. in Federated Gradient Averaging for Multi-Site Training with Momentum-Based Optimizers (eds Remedios, S. W. et al.) (Springer, 2020). source

      Not checked in the free run.

    • cites [43] Xu, Y. et al. A collaborative online AI engine for CT-based COVID-19 diagnosis. Preprint at https://www.medrxiv.org/content/10.1101/2020.05.10.20096073v2 (2020).

      Not checked in the free run.

    • cites [8] Yang, D. et al. Federated semi-supervised learning for Covid region segmentation in chest ct using multi-national data from China, Italy, Japan. Med. Image Anal. 70, 101992 (2021). source

      Not checked in the free run.

    • cites [44] Raisaro, J. L. et al. SCOR: A secure international informatics infrastructure to investigate COVID-19. J. Am. Med. Inform. Assoc. 27, 1721–1726 (2020). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Validation results confirmed that the global model is robust and support the hypothesis that federated learning-trained models generalize across healthcare systems.

    “The validation results confirmed that the global model is robust, supporting our hypothesis that FL trained models are generalisable across healthcare systems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The EXAM score can be used in the same way as CORISK for patient triage.

    “The EXAM score can be used in the same way.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The CORISK thresholds derived at MGB are applicable to other populations.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    In federated learning, data governance remains local, reducing privacy concerns, and only model weights or gradients are communicated between client-sites and the federated server.

    “Governance of data for FL is maintained locally, alleviating privacy concerns, with only model weights or gradients communicated between the client-sites and the federated server38,39.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [38] Ma, C. et al. On safeguarding privacy and security in the framework of federated learning. IEEE Netw. 34, 242–248 (2020). source

      Not checked in the free run.

    • cites [39] Brisimi, T. S. et al. Federated learning of predictive models from federated Electronic Health Records. Int. J. Med. Inform. 112, 59–67 (2018). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The global federated model EXAM was more robust and achieved better results on individual sites than any model trained only on local data.

    “The global FL model, EXAM, proved to be more robust and achieved better results on individual sites than any model that was trained only on local data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Because EXAM used cases with unknown SARS-CoV-2 status, it could provide input before RT-PCR results and be useful in real-life clinical settings.

    “We also used cases with unknown SARS-COV-2 status, and so the model could provide input to the physician ahead of receiving an RT-PCR test result, making it useful for a real-life clinical setting.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The study involved multiple sites across four continents under different regulatory bodies, which holds promise for expedited provision to different regulated markets.

    “Our study involved multiple sites across four continents and under the oversight of different regulatory bodies, and thus holds the promise of being provided to different regulated markets in an expedited way.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Like other machine learning models, EXAM is limited by the quality of its training data.

    “Similar to other machine learning models, EXAM is limited by the quality of the training data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A patient at lower risk of needing invasive oxygen therapy may be placed in a less intensive care setting or discharged for self-monitoring.

    “In contrast, a patient who is at a lower risk of requiring invasive oxygen therapy may be placed in a less intensive care setting such as a regular ward or even released from the ED for continued self-monitoring at home65.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [65] Kilaru, A. S. et al. Return hospital admissions among 1419 COVID-19 patients discharged from five U.S. emergency departments. Acad. Emerg. Med. 27, 1039–1042 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Diversity across 20 sites may have mitigated the risk of population drift due to different disease phases.

    “We believe that due to the diversity across 20 sites, this risk may have been mitigated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    EXAM is not approved by any regulatory agency and should be used only for research purposes.

    “Of note, the model is not approved by any regulatory agency at this time, and it should only be used for research purposes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Encryption algorithms such as homomorphic encryption maintain performance but may greatly increase message sizes and training time.

    “The encryption algorithms, such as Homomorphic Encryption60 shall maintain the performance but may significantly increase the message sizes and training time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [60] Zhang, C. et al. BatchCrypt: efficient homomorphic encryption for cross-silo federated learning. In Proc. 2020 USENIX Annual Technical Conference, ATC 2020, 493–506 (2020).

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Most AI models, including the CDS model, have been trained and validated on narrow, non-diverse data, which can lead to overfitting and lower generalizability.

    “To date, most AI models, including the aforementioned CDS model, have been trained and validated on ‘narrow’ data that often lacks in diversity31,32, potentially resulting in over-fitting and lower generalizability.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [31] Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25, 24–29 (2019). source

      Not checked in the free run.

    • cites [32] Cahan, E. M., Hernandez-Boussard, T., Thadaney-Israni, S. & Rubin, D. L. Putting the data before the algorithm in big data addressing personalized healthcare. NPJ Digit. Med. 2, 78 (2019). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The collected data reflect varied incidence rates, making the encountered population momentum more diverse.

    “The data collected reflects varied incidence rates, and thus the ‘population momentum’ we encountered is more diverse.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    The training data from the 20 participating institutes remained under their custody, was not shared with other institutions or the federated server, and is not publicly available.

    “The dataset from the 20 institutes that participated in this study remains under their custody. This data was used for training at each of the local sites and was not shared with any of the other participant institutions or with the Federated Server, and it is not publicly available.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The most established form of federated learning is the Federated Averaging algorithm or variations thereof.

    “Arguably, the most established form of FL is implementing the Federated Averaging algorithm proposed by McMahan et al72, or variations thereof.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites McMahan et al72

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The CDS model output a score called CORISK that corresponded to oxygen support requirements and could aid front-line clinicians in triaging patients.

    “The output of the CDS model was a score, termed ‘CORISK’27 that corresponded to oxygen support requirements, and that could aid in triaging patients by front-line clinicians28+30.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [27] Buch, V. et al. Development and validation of a deep learning model for prediction of severe outcomes in suspected COVID-19 Infection. Preprint at https://arxiv.org/abs/2103.11269 (2021).

      Not checked in the free run.

    • cites 28+30

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Differential privacy algorithms provide good protection but may weaken model performance.

    “While differential privacy algorithms36,48,49 show good protection, they may weaken the model’s performance.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [36] Rieke, N. et al. The future of digital health with federated learning. NPJ Dig. Med. 3, 119 (2020). source

      Not checked in the free run.

    • cites [48] Zhu, L., Liu, Z. & Han, S. in Advances in Neural Information Processing Systems 32 (eds Wallach, H. et al.) 14774–14784 (Curran Associates, Inc., 2019).

      Not checked in the free run.

    • cites [49] Kaissis, G. A., Makowski, M. R., Rückert, D. & Braren, R. F. Secure, privacy-preserving and federated machine learning in medical imaging. Nat. Mach. Intell. 2, 305–311 (2020). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The federated learning system used secure SSL encryption for communication between clients and the central server.

    “Each client site learns locally, and shares model weight updates with a central server that aggregates contributions using secure SSL encryption and communication protocols.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    All code, software, and trained models used in this study are publicly available on NVIDIA NGC.

    “All code and software used in this study are publicly available on NGC.”
    • Also stated in: § Code availability · plain
      “The trained models, data preparation guidelines, code for training, validating testing the model, readme file, installation guideline, and license files are publicly available at NVIDIA NGC61: https://ngc.nvidia.com/catalog/models/nvidia:med:clara_train_covid19_exam_ehr_xray”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61] Nvidia NGC Catalog: COVID-19 Related Models. https://ngc.nvidia.com/catalog/models?orderBy=scoreDESC&pageNumber=0&query=covid&quickFilter=models&filters (2020).

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The global federated model outperforming local and large-dataset models on all three prediction tasks indicates substantial benefit for small client-sites participating in federated collaborations.

    “The finding, that these two approaches were outperformed on all three prediction tasks by the global FL model, indicate that the benefit for client-sites with small datasets arising from participation in FL collaborations is substantial.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The consistent improvement was due to a larger and more diverse dataset, use of standardizable inputs, and avoidance of clinical impressions or reported symptoms.

    “We believe that consistent improvement was achieved not only due to larger, but also a more diverse data set, the use of data inputs that can be standardized and avoidance of clinical impressions / reported symptoms.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Unlike most published models focused on COVID-19 diagnosis or mortality, EXAM predicted oxygen requirements with implications for patient management.

    “Unlike the majority of the publications, focused on diagnosis of COVID-19 or predicting mortality, we predicted oxygen requirements that have implications for patient management.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Because data were not centralized, they are not readily accessible, limiting future analysis beyond what was derived and collected.

    “As data was not centralized, it is not readily accessible. Given that, any future analysis of the results, beyond what was derived and collected, is limited.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    EXAM's imaging input is used in common practice, unlike chest CT, which is a non-consensual diagnostic modality.

    “The model’s imaging input is used in common practice, in contrast with models that use chest Computed Tomography (CT), a non-consensual diagnostic modality.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    A critical decision in emergency evaluation is whether a patient is likely to need invasive or resource-limited interventions.

    “A critical decision made during the evaluation of a patient at the initial point of care or the ED, is whether the patient is likely to require more invasive or resource-limited countermeasures or interventions”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [64] Galloway, J. B. et al. A clinical risk score to identify patients with COVID-19 at high risk of critical care admission or death: an observational cohort study. J. Infect. 81, 282–288 (2020). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Federated learning carries potential risks of model inversion or reconstruction of training images from model gradients.

    “For FL, there remains the potential risk of model ‘inversion’47 or even the reconstruction of training images from the model gradients themselves48.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [47] Fredrikson, M., Jha, S. & Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proc. 22nd ACM SIGSAC Conference on Computer and Communications Security 1322–1333, https://doi.org/10.1145/2810103.2813677 (2015). source

      Not checked in the free run.

    • cites [48] Zhu, L., Liu, Z. & Han, S. in Advances in Neural Information Processing Systems 32 (eds Wallach, H. et al.) 14774–14784 (Curran Associates, Inc., 2019).

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The benefit for small sites likely reflects federated learning's ability to capture more diversity than local training and to reduce bias from homogeneous populations.

    “This is likely a reflection of FL’s ability to capture more diversity than local training, and to mitigate the bias present in models trained on a homogenous population.”
    Quote in the paperSubstantial claim
44 of 44

  • The authors state this as an assumption.

    Oxygen delivered to a patient is equivalent to the patient's oxygen need.

    “It is assumed that oxygen delivered to the patient equates with the oxygen need.”

  • The paper relies on this without support.

    In the early phase of the COVID-19 pandemic, many patients received high-flow oxygen prophylactically regardless of their oxygen need.

    “However, in the early phase of the COVID-19 pandemic, many patients were provided high flow oxygen prophylactically, regardless of their oxygen need.”

  • The paper relies on this without support.

    The clinical labels for oxygen support requirements accurately reflect the patients' true clinical states.

    “The final output was a continuous value in the range of 0 – 1 for both the 24 hour and 72-hour predictions, corresponding to the labels described above”

  • The paper relies on this without support.

    The test datasets at each site are independent and representative of the target populations.

    “After the FL training finishes, the best local models and the best global model are automatically shared with all client-sites and evaluated on their local test data.”

  • The paper relies on this without support.

    The CORISK thresholds derived at MGB are applicable to other populations.

    “The EXAM score can be used in the same way.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Federated learning is particularly helpful in osteoarthritis imaging because it uses several datasets reflecting worldwide diversity in demographics, imaging modalities, and disease presentations.

    “Because federated learning uses several datasets reflecting worldwide diversity in demographics, imaging modalities, and disease presentations, it is particularly helpful in osteoarthritis imaging [180].”

    From Biomarker-Guided Imaging and AI-Augmented Diagnosis of Degenerative Joint Disease (2025) · cites this paper as [180]

    The Discussion says federated learning can capture more diversity than local training, and Methods > Data Collection describes demographic differences and variation in chest X-ray acquisition across globally distributed sites. The study concerns COVID-19 and uses chest X-rays; it does not establish a particular benefit for osteoarthritis imaging or demonstrate the use of multiple imaging modalities. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Federated learning provides an alternative paradigm for accurate and generalizable machine learning by only sharing numerical model updates from decentralized data.

    “Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates.”

    From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [16]

  • Compared with existing real-world federated learning studies, this study's use-case is larger in scale and substantially more complex because it addresses a multi-parametric multi-class problem with reference standards requiring expert manual annotation and requires harmonized preprocessing for MRI acquisition differences.

    “The extended global footprint and the task complexity are what sets this study apart from current literature, since it dealt with a multi-parametric multi-class problem with reference standards that require expert clinicians following an involved manual annotation protocol, rather than simply recording a categorical entry from medical records16,32.”

    From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [16]

  • The governance insights from this study differ from previous literature that describes federated learning studies that were smaller in scale and involved simpler tasks.

    “These insights differ from previous literature that describes studies that were smaller in scale and involved simpler tasks16,32.”

    From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [16]

  • EXAM demonstrated improved average AUC and broader applicability compared to models trained at only one site.

    “demonstrating an improved average AUC and a broader applicability in contrast to models trained at only one site.”

    From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [56]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting counts of EMR input features

    Minor

    The Results section says the model uses 19 EMR features, while the Methods section says it uses 21. Both statements describe the model's inputs.

  • Five local sites but only four F-statistics

    Minor

    The statistical analysis says it reports F-values for five local sites, but the accompanying list contains only four values.

2 of 2

Cites

Flagged first, then by citations

3 of the 68 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Zhu, L., Liu, Z. & Han, S. in Advances in Neural Information Processing Systems 32 (eds Wallach, H. et al.) 14774–14784 (Curran Associates, Inc., 2019).
  2. Abadi, M. et al. TensorFlow: asystem for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), USENIX Association 265–283 (2016).
  3. Hsieh, K., Phanishayee, A., Mutlu, O. & Gibbons, P. B. The non-IID data quagmire of decentralized machine learning. In Proc. 37th International Conf. Machine Learning PMLR 119 (2020).

Cited by

Sorted by citations

Related

Authors
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., Wang, C.-H., Hsu, C.-N., Lee, C. K., Ruan, P., Xu, D., Wu, D., Huang, E., Kitamura, F. C., Lacey, G., de Antônio Corradi, G. C., Nino, G., Shin, H.-H., Obinata, H., Ren, H., Crane, J. C., Tetreault, J., Guan, J., Garrett, J. W., Kaggie, J. D., Park, J. G., Dreyer, K., Juluru, K., Kersten, K., Rockenbach, M. A. B. C., Linguraru, M. G., Haider, M. A., AbdelMaseeh, M., Rieke, N., Damasceno, P. F., e Silva, P. M. C., Wang, P., Xu, S., Kawano, S., Sriswasdi, S., Park, S. Y., Grist, T. M., Buch, V., Jantarabenjakul, W., Wang, W., Tak, W. Y., Li, X., Lin, X., Kwon, Y. J., Quraini, A., Feng, A., Priest, A. N., Turkbey, B., Glicksberg, B., Bizzo, B., Kim, B. S., Tor-Díez, C., Lee, C.-C., Hsu, C.-J., Lin, C., Lai, C.-L., Hess, C. P., Compas, C., Bhatia, D., Oermann, E. K., Leibovitz, E., Sasaki, H., Mori, H., Yang, I., Sohn, J. H., Murthy, K. N. K., Fu, L.-C., de Mendonça, M. R. F., Fralick, M., Kang, M. K., Adil, M., Gangai, N., Vateekul, P., Elnajjar, P., Hickman, S., Majumdar, S., McLeod, S. L., Reed, S., Gräf, S., Harmon, S., Kodama, T., Puthanakit, T., Mazzulli, T., de Lavor, V. L., Rakvongthai, Y., Lee, Y. R., Wen, Y., Gilbert, F. J., Flores, M. G., & Li, Q.
Affiliations
  • Harvard University (US)
  • Nvidia (United States) (US)
  • VA San Diego Healthcare System (US)
  • Icahn School of Medicine at Mount Sinai (US)
  • National Institutes of Health (US)
  • Tri-Service General Hospital (TW)
  • DASA (Brazil) (BR)
  • Memorial Sloan Kettering Cancer Center (US)
  • Self-Defense Forces Central Hospital (JP)
  • University of California, San Francisco (US)
  • University of Wisconsin–Madison (US)
  • University of Cambridge (GB)
  • Yeungnam University College (KR)
  • Children's National (US)
  • University of Toronto (CA)
  • Lunenfeld-Tanenbaum Research Institute (CA)
  • Chulalongkorn University (TH)
  • Kyungpook National University (KR)
  • Mass General Brigham (US)
  • National Taiwan University (TW)
  • Daegu Catholic University (KR)
  • National Defense Medical Center (TW)
  • National Health Insurance Administration (TW)
  • New York University (US)
  • Sinai Health System (CA)
  • University Health Network (CA)
Corresponding author
Flores, Mona G.
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2021-09-15
Volume
27
Issue
10
Pages
1735-1743
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-021-01506-3
PMC
PMC9157510
OpenAlex
W3200840849
Semantic Scholar
99b056a04fd4f98cad8b9987ff8d205fe62d786e
Cited by
842
References
68
Citations by year
2026: 194 · 2025: 166 · 2024: 150 · 2023: 197 · 2022: 122 · 2021: 10 · 2020: 1
FWCI
42.32
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, other-oa)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
  • Europe PMC (PubMed Central) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • Foundation for the National Institutes of Health
  • Doris Duke Charitable Foundation
  • Colgate-Palmolive Company
  • Brigham and Women's Hospital
  • Massachusetts General Hospital
  • American Association for Dental, Oral, and Craniofacial Research
  • Nvidia
  • National Center for Theoretical Sciences
  • Children's National Hospital
  • Lunit
  • Cancer Research UK (26884)
  • National Institute for Health and Care Research (NF-SI-0515-10067)
  • Department of Health and Social Care
  • University of Cambridge
  • Chulalongkorn University
  • Cambridge University Hospitals
  • Ministry of Science and Technology, Taiwan
  • Centre d'Imagerie BioMédicale
  • National Taiwan University
  • National Institutes of Health (COVID-19)
  • Genentech
  • University of California, San Francisco
  • National Health Insurance Administration
  • Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital
  • Engineering and Physical Sciences Research Council (EP/N014588/1)
  • Faculty of Medicine, Chulalongkorn University
  • NIHR Cambridge Biomedical Research Centre
  • National Heart, Lung, and Blood Institute (R01HL141237, R42HL145669)
  • National Cancer Institute (P30CA008748, ZIDBC011242)
  • National Institute of Allergy and Infectious Diseases (COVID-19)
  • U.S. National Library of Medicine (R01LM013151)
  • NIH Clinical Center (ZIACL040015)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
federated learning, COVID-19 pandemic, clinical outcome prediction, chest radiography, electronic health records, mechanical ventilation prediction, mortality prediction, model generalization, vital signs, laboratory findings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review